ptg16091132

The Go

Programming

Language

ptg16091132

This page intentionally left blank

ptg16091132

The Go

Programming

Language

Alan A. A. Donovan

Google Inc.

ptg16091132

Brian W. Kernighan

Princeton University

New York • Boston • Indianapolis • San Francisco

Toronto • Montreal • London • Munich • Paris • Madrid

Capetown • Sydney • Tokyo • Singapore • Mexico City

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals.

Th

e authors and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein.

For information about buying this title in bulk quantities, or for special sales opportunities (which may include electronic versions; custom cover designs; and content particular to your business, train-ing goals, marketing focus, or branding interests), please contact our corporate sales department at corpsales@pearsoned.com or (800) 382-3419.

For government sales inquiries, please contact governmentsales@pearsoned.com.

For questions about sales outside the United States, please contact international@pearsoned.com.

ptg16091132

Visit us on the Web: informit.com/aw

Library of Congress Control Number: 2015950709

Copyright © 2016 Alan A. A. Donovan & Brian W. Kernighan All rights reserved. Printed in the United States of America. Th is publication is protected by copyright,

and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. To obtain permission to use material from this work, please submit a written request to Pearson Education, Inc., Permissions Department, 200 Old Tappan Road, Old Tappan, New Jersey 07675, or you may fax your request to (201) 236-3290.

Front cover: Millau Viaduct, Tarn valley, southern France. A paragon of simplicity in modern engi-neering design, the viaduct replaced a convoluted path from capital to coast with a direct route over the clouds. © Jean-Pierre Lescourret/Corbis.

Back cover: the original Go gopher. © 2009 Renée French. Used under Creative Commons Attribu-tions 3.0 license.

Typeset by the authors in Minion Pro, Lato, and Consolas, using Go, groff , ghostscript, and a host of other open-source Unix tools. Figures were created in Google Drawings.

ISBN-13: 978-0-13-419044-0

ISBN-10: 0-13-419044-0

Text printed in the United States on recycled paper at RR Donnelley in Crawfordsville, Indiana.

First printing, October 2015

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

Fo r Leil a an d Me g

ptg16091132

This page intentionally left blank

ptg16091132

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

Cont ents

Pr eface xi

The Origins of Go

xii

The Go Pro jec t

xiii

Organizat ion of the Book

xv

ptg16091132

Wh ere to Find More Infor mat ion

xvi

Ac knowledgments xvii

1. Tutorial

1

1.1. Hel lo, Wor ld

1

1.2. Command-L ine Arguments 4

1.3. Finding Dup lic ate Lines 8

1.4. Animated GIFs 13

1.5. Fetching a URL 15

1.6. Fetching URLs Con cur rently

17

1.7. A We b Server 19

1.8. Loose End s

23

2. Pro gram Str ucture

27

2.1. Names 27

2.2. Declarat ions 28

2.3. Var iables 30

2.4. Assig nments 36

2.5. Typ e Decl arat ions 39

2.6. Packages and Files 41

2.7. Scope 45

vii

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

viii

CONTENTS

3. Basic Data Typ es

51

3.1. Int egers 51

3.2. Float ing-Point Numbers 56

3.3. Complex Numbers 61

3.4. Boole ans

63

3.5. Str ings

64

3.6. Con stants

75

4. Com posite Typ es

81

4.1. Arrays 81

4.2. Slices 84

4.3. Maps 93

4.4. Str ucts

99

4.5. JSON

107

4.6. Text and HTML Templ ates

113

5. Func tions 119

5.1. Function Declarat ions 119

5.2. Rec ursion

121

5.3. Multiple Retur n Values 124

5.4. Erro rs

127

5.5. Function Values 132

ptg16091132

5.6. Anony mou s Func tions 135

5.7. Var iadic Functions 142

5.8. Defer red Function Cal ls

143

5.9. Panic 148

5.10. Recov er

151

6. Metho ds

155

6.1. Met hod Declarat ions 155

6.2. Met hods wit h a Point er Receiver

158

6.3. Composing Typ es by Str uct Emb edding

161

6.4. Met hod Values and Expressions 164

6.5. Example: Bit Vec tor Typ e

165

6.6. Encapsulat ion

168

7. Interfaces

171

7.1. Int erfaces as Contrac ts

171

7.2. Int erface Typ es

174

7.3. Int erface Satisfac tion

175

7.4. Parsing Flags wit h flag.Value

179

7.5. Int erface Values 181

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

CONTENTS

ix

7.6. Sor ting wit h sort.Interface

186

7.7. The http.Handler Interface 191

7.8. The error Interface 196

7.9. Example: Expression Evaluator

197

7.10. Typ e As ser tions 205

7.11. Discriminat ing Error s with Typ e As ser tions 206

7.12. Quer ying Beh avior s with Int erface Typ e As ser tions 208

7.13. Typ e Sw itch es

210

7.14. Example: Token-B ased XML Decoding

213

7.15. A Fe w Wo rds of Adv ice 216

8. Gor o utines and Channels 217

8.1. Goroutines

217

8.2. Example: Con cur rent Clo ck Ser ver 219

8.3. Example: Con cur rent Ech o Server 222

8.4. Channel s

225

8.5. Looping in Paral lel

234

8.6. Example: Con cur rent Web Craw ler

239

8.7. Multiplexing wit h select

244

8.8. Example: Con cur rent Direc tor y Traversal

247

8.9. Cancellat ion

251

ptg16091132

8.10. Example: Chat Ser ver 253

9. Concurrency with Shared Vari ables

257

9.1. Race Con dit ion s

257

9.2. Mut ual Exc lusion: sync.Mutex

262

9.3. Read/Write Mut exes: sync.RWMutex

266

9.4. Memor y Sy nchro nizat ion

267

9.5. Lazy Initializat ion: sync.Once

268

9.6. The Race Detec tor 271

9.7. Example: Con cur rent Non-Blo cking Cache 272

9.8. Goroutines and Threads 280

10. Pack ages and the Go Tool

283

10.1. Int roduc tion

283

10.2. Imp ort Pat hs

284

10.3. The Package Declarat ion

285

10.4. Imp ort Declarat ions 285

10.5. Blank Imp orts

286

10.6. Packages and Naming

289

10.7. The Go Tool

290

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

x

CONTENTS

11. Testing 301

11.1. The go test To ol

302

11.2. Test Func tions 302

11.3. Cov erage

318

11.4. Benchmark Func tions 321

11.5. Profiling

323

11.6. Example Func tions 326

12. Reflecti on

329

12.1. Why Reflec tion?

329

12.2. reflect.Type and reflect.Value

330

12.3. Display, a Rec ursive Value Print er

333

12.4. Example: Enco ding S-E xpressions 338

12.5. Setting Var iables wit h reflect.Value

341

12.6. Example: Decoding S-E xpressions 344

12.7. Accessing Str uct Field Tags 348

12.8. Displaying the Met hods of a Typ e

351

12.9. A Wo rd of Caution 352

13. Low-L evel Pro gramming

353

13.1. unsafe.Sizeof, Alignof, and Offsetof

354

13.2. unsafe.Pointer

356

ptg16091132

13.3. Example: Deep Equivalence 358

13.4. Cal ling C Code wit h cgo

361

13.5. Another Word of Caution 366

Index

367

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

Pref ace

‘‘Go is an open sourc e prog rammi n g lang uage that mak es it easy to bui ld simpl e, rel iab le, an d efficient sof tware.’’ (Fro m the Go web sit e at golang.org) ptg16091132

Go was conceive d in Septemb er 2007 by Rob ert Gries emer, Rob Pike, and Ken Thomps on, all at Google, and was announced in Nov emb er 2009. The goals of the langu age and its accom-pany ing tools were to be expressive, efficient in bot h comp ilation and exe cut ion, and effec tive in writing reliable and robust programs.

Go bears a sur face simi lar ity to C and, like C, is a tool for prof essional programmers, achie ving maximum effe ct wit h minimum means. But it is much more than an updated version of C. It bor rows and adapts good ide as from many other langu ages, whi le avoiding features that have led to complexity and unreliable code. Its facilities for con cur rency are new and efficient, and its approach to dat a abstrac tion and obj e ct-oriente d prog ramming is unu sually flexible. It has aut omat ic memory management or garb age col lec tion.

Go is esp eci ally wel l suit ed for bui lding inf rastr ucture like net wor ked ser vers, and tools and systems for prog rammers, but it is truly a general-pur pos e language and find s us e in domains as divers e as graphics, mobile app lic ations, and machine learning. It has becom e popu lar as a repl acement for unt yped scr ipt ing langu ages because it bal ances expressiveness wit h safety : Go programs typic ally run faster than programs writt en in dynamic langu ages and suf fer far fe wer crashes due to unexp ected typ e er ror s.

Go is an open-s ource pro jec t, so source code for its compi ler, librar ies, and tools is fre ely avai lable to any one. Contr ibution s to the pro jec t come fro m an active worldw ide community. Go runs on Unix-li ke systems—Linux, Fre eBSD, OpenBSD, Mac OS X—and on Plan 9 and Micros oft Windows. Programs writt en in one of these env iro nments general ly wor k without mo dification on the others.

xi

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

xii

PREFACE

This book is meant to help you start using Go effec tive ly rig ht away and to use it wel l, tak ing full advantage of Go’s langu age features and stand ard librar ies to write clear, idiomatic, and efficient programs.

The Origins of Go

Like biolog ical species, successf ul langu ages beget offspring that incor porate the advantages of their ancestors; int erbre e ding som etimes leads to sur prising strengt hs; and, ver y occasionally, a radic al ne w fe ature arises wit hout pre cedent. We can lear n a lot about why a langu age is the way it is and what env iro nment it has been adapt ed for by looking at these influences.

The figure below shows the most imp ortant influences of earlier programming langu ages on the desig n of Go.

ptg16091132

Go is som etimes des crib ed as a ‘‘C-li ke langu age,’’ or as ‘‘C for the 21st century.’’ Fr om C, Go in her ite d its expression syntax, cont rol-flow statements, basic dat a types, cal l-by-value parameter passing, point ers, and above all, C’s emp hasis on programs that compi le to efficient machine code and cooperate natural ly wit h the abstrac tions of cur rent operat ing systems.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

THE ORIGINS OF GO

xiii

But there are other ancestors in Go’s fami ly tre e. One maj or stream of influence comes fro m languages by Nik laus Wir th, beg inning wit h Pascal. Modula-2 inspired the package con cept.

Ob ero n eliminated the distinc tion bet ween module int erface files and module imp lementation files. Obero n-2 influence d the syntax for packages, imports, and declarat ions, and Obj e ct Ob ero n prov ide d the syntax for met hod declarat ions.

Anot her lineage among Go’s ancestors, and one that makes Go distinc tive among recent prog ramming langu ages, is a sequence of lit tle-k now n research langu ages deve lop ed at Bel l Labs, all ins pired by the con cept of commu nicating sequent ial pro cesses (CSP) fro m To ny Ho are’s seminal 1978 pap er on the found ation s of con cur rency. In CSP, a program is a paral lel comp osition of processes that have no share d st ate; the pro cesses communic ate and synchro-nize using channel s. But Hoare’s CSP was a for mal langu age for des cribing the fundament al concepts of con cur rency, not a programming langu age for writing exe cut able programs.

Rob Pike and others began to exp eriment wit h CSP imp lementation s as actu al langu ages. The firs t was cal le d Sque ak (‘‘A langu age for communic ating wit h mice’’), which provide d a langu age for handling mou se and key board events, wit h st atically cre ate d ch annel s. This was fo llowe d by Newsque ak, which offered C-li ke statement and expression syntax and Pas cal-li ke type not ation. It was a purely functional langu age wit h garb age col lec tion, again aimed at managing key board , mous e, and window events. Channel s became firs t-class values, dynami-cally cre ate d and storable in variables.

ptg16091132

The Plan 9 operat ing system car r ied these ide as forward in a langu age cal le d Alef. Alef tried to make Newsque ak a viable system programming langu age, but its omission of garb age colle ction made conc urrency too painf ul.

Ot her cons tructions in Go show the influence of non-ancestral genes here and there; for example iota is loosely fro m APL, and lexic al scop e with neste d func tions is fro m Scheme (and most langu ages since). Here too we find nove l mu tat ions. Go’s innovat ive slices provide dy namic arrays wit h efficient random access but als o permit sop histicate d sh aring ar rangements reminiscent of lin ked lists. And the defer st atement is new wit h Go.

The Go Project

Al l prog ramming langu ages reflec t the programming phi losop hy of their creators, which often includes a significant component of reaction to the perceive d shortcomings of earlier langu ages. The Go pro jec t was bor ne of frust rat ion wit h several sof tware systems at Google that were suf fer ing fro m an explosion of complexity. (This pro blem is by no means unique to Go ogle.)

As Rob Pike put it, ‘‘comp lexity is multiplic ative’’: fixing a pro blem by mak ing one par t of the system more complex slowly but surely adds complexity to other par ts. Wi th con stant pres-sure to add features and opt ion s and configurat ions, and to ship code quickly, it’s easy to neglec t simplicity, even thoug h in the lon g run simplicity is the key to good sof tware.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

xiv

PREFACE

Simp licity requires more wor k at the beg inning of a pro jec t to reduce an idea to its essence and more dis cip line over the lifet ime of a pro jec t to distinguish good changes fro m bad or per nicious ones. Wit h sufficient effor t, a good change can be accommodated wit hout compromis-ing what Fre d Brooks cal le d the ‘‘conceptu al int egr ity’’ of the desig n but a bad change cannot, and a per nicious change trades simplicity for its shallow cou sin, conv enience. Only through simplicity of desig n can a system remain stable, sec ure , and coh erent as it grows.

The Go pro jec t includes the langu age its elf, its tools and stand ard librar ies, and last but not le ast, a cultural agenda of radic al simplicity. As a recent hig h-leve l language , Go has the benefit of hindsig ht, and the basics are don e we ll: it has garbage col lec tion, a package system, firs t-cl ass functions, lexic al scop e, a system cal l interface, and immut able str ings in which text is general ly encoded in UTF-8. But it has comp arat ive ly few features and is unlikely to add more . Fo r inst ance, it has no imp licit numer ic conv ersions, no con str uctor s or destr uctor s, no op erator overloading, no defau lt parameter values, no inher itance, no gener ics, no exception s, no macros, no function annot ation s, and no thread-lo cal storage. The langu age is mature and st able, and guarante es backwards compatibi lit y: older Go programs can be compi led and run with newer versions of compi lers and stand ard librar ies.

Go has enough of a typ e system to avoid most of the careless mistakes that plague programmers in dynamic langu ages, but it has a simpler typ e system than comparable typ ed langu ages.

This appro ach can som etimes lead to isolate d pockets of ‘‘untyped’’ prog ramming wit hin a broader framewor k of typ es, and Go programmers do not go to the lengt hs that C++ or ptg16091132

Haskel l prog rammers do to express safet y prop erties as typ e-bas ed pro ofs. But in prac tice Go gives programmers much of the safet y and run-t ime per for mance benefits of a rel ative ly st ron g type system wit hout the burden of a complex one.

Go encourages an awareness of contemp orar y comp uter system desig n, partic ularly the importance of locality. Its bui lt-in dat a types and most librar y data str uctures are craf ted to work natural ly wit hout explicit initializat ion or imp licit con str uctor s, so rel ative ly few memor y al location s and memor y wr ites are hidden in the code. Go’s aggregate typ es (str ucts and ar rays) hold their elements direc tly, requir ing less storage and fewer allo cat ions and point er indirec tion s than langu ages that use indirec t fields. And since the moder n comp uter is a paral lel machine, Go has conc urrency features bas ed on CSP, as mention ed earlier. The var iable-size stacks of Go’s lig htweig ht threads or goro utines are initial ly small enoug h that creating one goro utine is che ap and cre ating a million is prac tic al.

Go’s stand ard librar y, often descr ibed as coming wit h ‘‘batt eries include d,’’ prov ides cle an building blo cks and APIs for I/O, text pro cessing, graphics, cryptography, net wor king, and dist ribut ed app lic ations, wit h supp ort for many stand ard file for mats and pro tocol s. The librar ies and tools make extensive use of convent ion to reduce the need for configurat ion and explanation, thu s simplif ying program logic and mak ing diverse Go programs more simi lar to each other and thu s easier to learn. Pro jec ts bui lt using the go to ol use only file and identifier names and an occ asional speci al comment to deter mine all the librar ies, exec utables, tests, benchmarks, examples, platfor m-specific var iants, and documentation for a proj e ct; the Go source its elf cont ains the bui ld sp ecification.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

THE GO PROJECT

xv

Or ganization of the Book

We assume that you have programmed in one or more other langu ages, whether comp ile d li ke C, C++, and Java, or int erprete d li ke Pyt hon, Ruby, and JavaS crip t, so we won’t spell out everything as if for a tot al beg inner. Sur face sy ntax will be fami liar, as will var iables and con stants, expressions, control flow, and functions.

Chapter 1 is a tut orial on the basic cons tructs of Go, int roduce d thro ugh a dozen programs for ever yday tasks like reading and writing files, for matting text, creating images, and communi-cating wit h Internet clients and ser vers.

Chapter 2 des crib es the str uctural elements of a Go program—de clarat ions, var iables, new types, packages and files, and scope. Chapt er 3 dis cusses numbers, boole ans, str ings, and const ants, and explains how to pro cess Unico de. Chapt er 4 des crib es composite typ es, that is, types bui lt up fro m simpler ones using arrays, maps, str ucts, and sli ces, Go’s appro ach to dy namic lists. Chapt er 5 cov ers functions and dis cusses error handling, panic and recover, and the defer st atement.

Chapters 1 through 5 are thu s the basics, things that are par t of any mainst ream imp erat ive language . Go’s syntax and sty le som etimes dif fer fro m ot her langu ages, but most programmers will pick them up quickly. The remaining chapt ers focus on topics where Go’s appro ach is less convent ion al: met hods, interfaces, conc urrency, packages, testing , and reflec tion.

ptg16091132

Go has an unusual appro ach to obj e ct-oriente d prog ramming. There are no class hierarchies, or indeed any class es; comp lex obj e ct beh avior s are cre ate d from simpler ones by composition, not inher itance. Met hods may be ass oci ated wit h any user-define d type, not just str uctures, and the rel ation ship bet ween con crete typ es and abstrac t types (interfaces) is imp licit, so a concrete typ e may sat isf y an interface that the typ e’s desig ner was unaware of. Met hods are covere d in Chapt er 6 and int erfaces in Chapt er 7.

Chapter 8 presents Go’s appro ach to con cur rency, which is bas ed on the ide a of communic ating sequential pro cesses (CSP), embodie d by goroutines and channel s. Chapt er 9 explains the more tradition al asp ects of con cur rency bas ed on share d var iables.

Chapter 10 des crib es packages, the mech anism for organizing librar ies. This chapt er als o shows how to make effec tive use of the go to ol, which provides for compi lat ion, testing , benchmarking , prog ram formatting , do cumentation, and many other tasks, all wit hin a single command.

Chapter 11 deals wit h test ing , where Go takes a not ably lig htweig ht appro ach, avoiding abstrac tion-l aden framewor ks in favor of simple librar ies and tools. The testing librar ies prov ide a found ation atop which more complex abstrac tions can be bui lt if necessary.

Chapter 12 dis cusses reflec tion, the abi lit y of a prog ram to examine its own represent ation during exe cut ion. Reflec tion is a pow erful tool, thoug h on e to be used caref ully; this chapt er explains finding the rig ht bal ance by showing how it is used to imp lement som e important Go librar ies. Chapter 13 explains the gor y det ails of low-le vel programming that uses the unsafe package to step aro und Go’s typ e system, and when that is appro priate.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

xvi

PREFACE

Each chapter has a number of exercis es that you can use to test your underst anding of Go, and to explore extensions and alt ernat ives to the examples fro m the book.

Al l but the most trivial code examples in the book are avai lable for dow nlo ad from the public Git rep ository at gopl.io. Each example is identified by its package imp ort pat h and may be conv eniently fetch ed, bui lt, and ins tal le d using the go get command. You’ll need to cho ose a direc tor y to be your Go wor ksp ace and set the GOPATH enviro nment var iable to point to it.

The go to ol will cre ate the direc tor y if necessary. For example: $ export GOPATH=$HOME/gobook

# choose workspace directory

$ go get gopl.io/ch1/helloworld

# fetch, build, install

$ $GOPATH/bin/helloworld

# run

Hello, BF

To run the examples, you will need at least version 1.5 of Go.

$ go version

go version go1.5 linux/amd64

Fo llow the ins tructions at https://golang.org/doc/install if the go to ol on your computer is older or missing.

Where to Find More Information

ptg16091132

The best source for more infor mat ion about Go is the offici al we b site, https://golang.org,

which provides access to the documentation, including the Go Programmi n g Lang uage Specificati on, stand ard packages, and the like. There are als o tutorials on how to write Go and how to write it wel l, and a wide variet y of online text and vide o resources that will be valuable complements to this book. The Go Blog at blog.golang.org pu blishes som e of the best writing on Go, wit h ar ticles on the state of the langu age, plans for the fut ure , reports on conferences, and in-depth explanation s of a wide variet y of Go-rel ate d topics.

One of the most useful asp ects of online access to Go (and a reg rettable limitation of a pap er book) is the abi lit y to run Go programs fro m the web pages that des crib e them. This function ality is provide d by the Go Playg round at play.golang.org, and may be emb edde d within other pages, such as the home page at golang.org or the documentation pages ser ved by the godoc to ol.

The Playg round makes it convenient to per for m simple exp eriments to che ck one’s underst anding of syntax, semantics, or librar y packages wit h short programs, and in many ways takes the place of a re ad-e val-print loop (REPL) in other langu ages. Its persistent URLs are great for sharing snipp ets of Go code wit h ot hers, for rep orting bugs or mak ing sug gestion s.

Built atop the Playg round, the Go Tour at tour.golang.org is a sequence of short int erac tive lessons on the basic ideas and con str uctions of Go, an orderly wal k thro ugh the langu age.

The primary shortcoming of the Playg round and the Tour is that the y al low only stand ard librar ies to be imp orted, and many librar y fe atures—networ king, for example—are restr icted

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

WHERE TO FIND MORE INFORMATION

xvii

for prac tic al or sec urity reasons. The y also require access to the Int ernet to compi le and run each program. So for more elaborate exp eriments, you will have to run Go programs on your ow n comp uter. For tunately the dow nlo ad process is straig htfor ward, so it should not take more than a few minut es to fetch the Go distr ibution fro m golang.org and start writing and running Go programs of your own.

Since Go is an open-s ource pro jec t, you can read the code for any typ e or function in the standard librar y on line at https://golang.org/pkg; the same code is par t of the dow nlo ade d dist ribut ion. Use this to figure out how som ething wor ks, or to answer question s ab out det ails, or merely to see how exp erts write really good Go.

Acknowledgments

Rob Pike and Russ Cox, core members of the Go team, read the manus cript several times wit h great care; their comments on everything fro m word choice to overal l st ruc ture and organization have been invaluable. Whi le prep aring the Jap anes e transl ation, Yoshi ki Shibata went far beyond the cal l of duty; his met iculou s eye spott ed numerou s inconsistencies in the Eng lish text and error s in the code. We gre atly appre ciate thoro ugh revie ws and critical comments on the ent ire manus cript fro m Br ian Goetz, Corey Kos ak, Ar nold Robbins, Josh Ble e cher Sny der, and Peter Weinberger.

ptg16091132

We are indebte d to Sameer Ajmani, Itt ai Bal aban, Dav id Crawshaw, Billy Don ohue, Jon athan Feinberg , Andrew Gerrand, Rob ert Gries emer, John Linder man, Minux Ma, Bryan Mills, Bal a Nataraj an, Cosmos Nicolaou , Paul Stanifor th, Nigel Tao, and Howard Trickey for many helpf ul sug gestion s. We als o than k David Brai lsford and Raph Levien for typ esetting adv ice.

Our editor Greg Doench at Addison-Wesle y got the bal l ro lling originally and has been con-tinuou sly helpf ul ever since. The AW pro duc tion team—Jo hn Fuller, Day na Isley, Julie Nahi l, Chut i Pras ertsith, and Barb ara Wo o d—has been outst anding; author s could not hop e for better sup por t.

Alan Don ovan wishes to thank: Sameer Ajmani, Chr is Demet riou, Walt Drummon d, and Reid Tatge at Google for allow ing him time to write; Steph en Don ovan, for his adv ice and timely encouragement; and above all, his wife Lei la Kazemi, for her unhesitating ent husi asm and unwaver ing sup por t for this pro jec t, despit e the lon g hours of distrac tion and abs ente eism from fami ly life that it ent ailed.

Br ian Ker nighan is deeply gratef ul to friends and col leagues for their pat ience and forbearance as he mov ed slowly along the pat h to underst anding, and esp eci ally to his wife Meg , who has been unfailingly sup por tive of book-w rit ing and so much els e.

Ne w Yo rk

Oc tob er 2015

This page intentionally left blank

ptg16091132

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

1

Tutorial

This chapt er is a tour of the basic comp onents of Go. We hop e to provide enough infor mat ion and examples to get you off the gro und and doing useful things as quickly as possible. The examples here, and indeed in the whole book, are aimed at tasks that you mig ht have to do in the real wor ld. In this chapt er we’ll try to give you a taste of the diversity of programs that one ptg16091132

mig ht write in Go, ranging fro m simple file pro cessing and a bit of graphics to con cur rent Internet clients and ser vers. We cer tain ly won’t explain everything in the firs t ch apt er, but studying such programs in a new langu age can be an effec tive way to get started.

Wh en you’re learning a new langu age, there’s a natural tendency to write code as you wou ld have writt en it in a langu age you already know. Be aware of this bias as you learn Go and try to avoid it. We’ve tried to illustrate and explain how to write good Go, so use the code here as a guide when you’re writing your own.

1.1. Hello, World

We’ll start wit h the now-t radition al ‘‘hel lo, wor ld’’ example, which app ears at the beg inning of The C Programmi n g Lang uage, publishe d in 1978. C is one of the most direc t influences on Go, and ‘‘hel lo, wor ld’’ illustrates a number of central ideas.

gopl.io/ch1/helloworld

package main

import "fmt"

func main() {

fmt.Println("Hello, BF")

}

1

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

2

CHAPTER 1.

TUTORIAL

Go is a compi led langu age. The Go toolchain conv erts a source program and the things it dep ends on int o inst ruc tions in the nat ive machine langu age of a computer. These tools are accessed through a single command cal le d go that has a number of sub command s. The simplest of these sub command s is run, which compi les the source code fro m on e or more source files whose names end in .go, lin ks it wit h librar ies, then runs the resulting exe cut able file.

(We will use $ as the command pro mpt throughout the book.) $ go run helloworld.go

No t surprisingly, this prints

Hello, BF

Go nat ive ly handles Unico de, so it can pro cess text in all the wor ld’s langu ages.

If the program is more than a one-shot exp eriment, it’s likely that you wou ld want to compi le it once and save the compi led result for later use. That is don e with go build: $ go build helloworld.go

This cre ates an exe cut able binar y file cal le d helloworld that can be run any time wit hout fur-ther pro cessing:

$ ./helloworld

Hello, BF

ptg16091132

We have lab ele d each sig nificant example as a reminder that you can obt ain the code fro m the book’s source code rep ository at gopl.io:

gopl.io/ch1/helloworld

If you run go get gopl.io/ch1/helloworld, it will fetch the source code and place it in the corresponding direc tor y. There’s more about this topic in Sec tion 2.6 and Sec tion 10.7.

Let’s now tal k ab out the program its elf. Go code is organize d into packages, which are simi lar to librar ies or modules in other langu ages. A package con sists of one or more .go source files in a single direc tor y that define what the package does. Each source file beg ins wit h a package de clarat ion, here package main, that states which package the file belon gs to, fol low ed by a list of other packages that it imp orts, and then the declarat ions of the program that are store d in that file.

The Go stand ard librar y has over 100 packages for common tasks like inp ut and out put, sorting, and text manipu lat ion. For ins tance, the fmt package contains functions for printing formatte d output and scanning inp ut. Println is one of the basic out put functions in fmt; it pr ints one or more values, sep arated by spaces, wit h a newline charac ter at the end so that the values appear as a single line of out put.

Package main is speci al. It defines a stand alone exe cut able program, not a librar y. Wit hin package main the func tion main is als o sp eci al—it’s where exe cut ion of the program beg ins.

Wh ate ver main do es is what the program does. Of course, main wi l l normal ly cal l up on function s in other packages to do much of the wor k, such as the function fmt.Println.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 1.1. HELLO, WORLD

3

We must tel l the compi ler what packages are needed by this source file; that’s the role of the import de clarat ion that fol lows the package de clarat ion. The ‘‘hel lo, wor ld’’ prog ram us es on ly one function fro m on e ot her package, but most programs will imp ort more packages.

Yo u mu st imp ort exac tly the packages you need. A prog ram wi l l not compi le if there are missing imp orts or if there are unnecessary ones. This str ict requirement pre vents references to unu sed packages fro m acc umulat ing as programs evo l ve.

The import de clarat ions must fol low the package de clarat ion. After that, a program consists of the declarat ions of functions, var iables, cons tants, and typ es (introduce d by the key words func, var, const, and type); for the most par t, the order of declarat ions does not matter. This prog ram is about as short as possible since it declares only one function, which in tur n calls on ly one other function. To save space, we will som etimes not show the package and import de clarat ions when present ing examples, but the y are in the source file and must be there to comp ile the code.

A function declarat ion con sists of the key word func, the name of the function, a parameter list (empt y for main), a result list (also emp ty here), and the body of the function—the statements that define what it does—enclos ed in braces. We’ll take a clos er lo ok at functions in Chapter 5.

Go does not require semicolons at the end s of statements or declarat ions, except where two or ptg16091132

more app ear on the same line. In effec t, newlines fol low ing cer tain tokens are converted int o semicolons, so where newlines are place d matt ers to pro per parsing of Go code. For ins tance, the opening brace { of the function must be on the same line as the end of the func de claration, not on a line by its elf, and in the expression x + y, a newline is per mitt ed after but not before the + op erator.

Go takes a stron g st ance on code for matting . The gofmt to ol rewrites code int o the stand ard format, and the go to ol’s fmt su bcommand app lies gofmt to all the files in the specified package, or the ones in the cur rent direc tor y by defau lt. All Go source files in the book have been run through gofmt, and you should get into the habit of doing the same for your own code.

Decl aring a stand ard for mat by fiat eliminates a lot of point less deb ate about trivia and, more importantly, enables a variet y of aut omated source code transfor mat ions that wou ld be infeasible if arbit rar y formatting were allow ed.

Many text editors can be configured to run gofmt each time you save a file, so that your source co de is always pro perly for matte d. A rel ate d to ol, goimports, addition ally manages the ins ertion and removal of imp ort declarat ions as needed. It is not par t of the stand ard distr ibution but you can obt ain it wit h this command:

$ go get golang.org/x/tools/cmd/goimports

Fo r most users, the usu al way to dow nlo ad and bui ld packages, run their tests, show their documentation, and so on, is wit h the go to ol, which we’ll look at in Sec tion 10.7.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

4

CHAPTER 1.

TUTORIAL

1.2. Command-Line Arguments

Most programs pro cess som e input to pro duce som e output; that’s prett y much the definition of computing . But how does a program get input dat a on which to operate? Som e prog rams generate their own data, but more often, input com es fro m an exter nal source: a file, a net wor k connec tion, the out put of another program, a user at a key board , command-line arguments, or the like. The next few examples will dis cuss som e of these alt ernat ives, starting wit h command-line arguments.

The os package provides functions and other values for dealing wit h the operat ing system in a pl atfor m-indep endent fashion. Command-line arguments are avai lable to a program in a var iable named Args that is par t of the os package; thu s its name any where outside the os package is os.Args.

The var iable os.Args is a sli ce of str ings. Slices are a fundament al not ion in Go, and we’ll tal k a lot more about them soon. For now, thin k of a slice as a dynamic ally size d sequence s of ar ray elements where indiv idu al elements can be accessed as s[i] and a contiguous subsequence as s[m:n]. The number of elements is given by len(s). As in most other programming langu ages, all indexing in Go uses half-open intervals that include the firs t index but exclude the last, because it simplifies logic. For example, the slice s[m:n], where 0 ≤ m ≤ n ≤

len(s)

ptg16091132

, contains n-m elements.

The firs t element of os.Args, os.Args[0], is the name of the command its elf; the other elements are the arguments that were present ed to the program when it started exe cut ion. A slice expression of the for m s[m:n] yields a slice that refers to elements m thro ugh n-1, so the elements we need for our next example are those in the slice os.Args[1:len(os.Args)]. If m or n is omitt ed, it defau lts to 0 or len(s) resp ectively, so we can abbreviate the desired slice as os.Args[1:].

Here’s an imp lementation of the Unix echo command, which prints its command-line arguments on a single line. It imp orts two packages, which are given as a parenthesize d list rat her than as individu al import de clarat ions. Eit her for m is legal, but convent ion ally the list for m is us ed. The order of imp orts doesn’t matter; the gofmt to ol sor ts the package names into alph abetical order. (Wh en there are several versions of an example, we will often number them so you can be sure of which one we’re tal king about.) gopl.io/ch1/echo1

// Echo1 prints its command-line arguments.

package main

import (

"fmt"

"os"

)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 1.2. COMMAND-LINE ARGUMENTS

5

func main() {

var s, sep string

for i := 1; i < len(os.Args); i++ {

s += sep + os.Args[i]

sep = " "

}

fmt.Println(s)

}

Comments beg in with //. All text fro m a // to the end of the line is comment ary for prog rammers and is ignored by the compi ler. By convent ion, we des crib e each package in a comment immediate ly pre ceding its package declarat ion; for a main package, this comment is on e or more complete sentences that des crib e the program as a whole.

The var de clarat ion declares two var iables s and sep, of typ e string. A var iable can be initialize d as par t of its declarat ion. If it is not explicitly initialize d, it is imp licitly initialize d to the zero value for its typ e, which is 0 for numer ic types and the emp ty str ing "" for str ings.

Thus in this example, the declarat ion imp licitly initializes s and sep to emp ty str ings. We’ll have more to say about var iables and declarat ions in Chapt er 2.

Fo r numb ers, Go provides the usu al arithmetic and log ical operator s. When app lie d to st rings, how ever, the + op erator concatenate s the values, so the expression sep + os.Args[i]

ptg16091132

repres ents the con catenat ion of the str ings sep and os.Args[i]. The statement we used in the program,

s += sep + os.Args[i]

is an assig nment statement that conc atenates the old value of s with sep and os.Args[i] and assig ns it back to s; it is equivalent to

s = s + sep + os.Args[i]

The operator += is an assig nment operator. Each arithmetic and log ical operator like + or * has a cor responding assig nment operator.

The echo prog ram could have print ed its out put in a loop one pie ce at a time, but this version instead bui lds up a str ing by rep eatedly app ending new text to the end. The str ing s st arts life empt y, that is, wit h value "", and each trip through the loop adds som e text to it; after the firs t it erat ion, a space is als o inserted so that when the loop is finishe d, there is one space bet ween each argument. This is a quadrat ic process that cou ld be costly if the number of arguments is large , but for echo, that’s unlikely. We’ll show a number of imp rov ed versions of echo in this ch apt er and the next that will deal wit h any real inefficiency.

The loop index variable i is declare d in the firs t part of the for lo op. The := sy mbol is par t of a sh ort var iab le declarati on, a statement that declares one or more var iables and gives them appropriate typ es based on the initializer values; there’s more about this in the next chapt er.

The increment statement i++ adds 1 to i; it’s equivalent to i += 1 which is in tur n equivalent to i = i + 1. There’s a cor responding decrement statement i-- that subtrac ts 1. Thes e are

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

6

CHAPTER 1.

TUTORIAL

st atements, not expressions as the y are in most langu ages in the C fami ly, so j = i++ is illegal, and the y are postfix only, so --i is not legal either.

The for lo op is the only loop statement in Go. It has a number of for ms, one of which is illustrated here:

for initialization; condition; post {

// zero or more statements

}

Parent hes es are never used aro und the three components of a for lo op. The braces are mand atory, how ever, and the opening brace must be on the same line as the post st atement.

The opt ion al initialization st atement is exe cut ed before the loop starts. If it is present, it mu st be a simp le statement, that is, a short var iable decl arat ion, an increment or assig nment st atement, or a function cal l. The condition is a boole an expression that is evaluate d at the beginning of each iterat ion of the loop; if it evaluates to true, the statements control led by the lo op are exe cut ed. The post st atement is exe cut ed after the body of the loop, then the con dition is evaluate d again. The loop end s when the con dit ion becom es false.

Any of these par ts may be omitt ed. If there is no initialization and no post, the semicolons may also be omitt ed:

ptg16091132

// a traditional "while" loop

for condition {

// ...

}

If the con dit ion is omitt ed ent ire ly in any of these for ms, for example in

// a traditional infinite loop

for {

// ...

}

the loop is infinite, thoug h lo ops of this for m may be ter minated in som e ot her way, like a break or return st atement.

Anot her for m of the for lo op iterates over a ra nge of values fro m a dat a type like a st ring or a slice. To illustrate, here’s a secon d version of echo: gopl.io/ch1/echo2

// Echo2 prints its command-line arguments.

package main

import (

"fmt"

"os"

)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 1.2. COMMAND-LINE ARGUMENTS

7

func main() {

s, sep := "", ""

for _, arg := range os.Args[1:] {

s += sep + arg

sep = " "

}

fmt.Println(s)

}

In each iterat ion of the loop, range produces a pair of values: the index and the value of the element at that index. In this example, we don’t need the index, but the syntax of a range lo op re quires that if we deal wit h the element, we must deal wit h the index too. One ide a would be to assig n the index to an obv iou sly temporar y var iable like temp and ignore its value, but Go do es not per mit unu sed local variables, so this wou ld result in a compi lat ion error.

The solut ion is to use the bl ank ident ifier, whose name is _ (t hat is, an underscore). The blank identifier may be used whene ver synt ax requires a variable name but prog ram log ic do es not, for ins tance to dis card an unwante d lo op index when we require only the element value. Most Go programmers wou ld likely use range and _ to write the echo prog ram as abov e, since the indexing over os.Args is imp licit, not explicit, and thu s easier to get rig ht.

This version of the program uses a short var iable decl arat ion to declare and initialize s and ptg16091132

sep, but we cou ld equ ally wel l have declare d the var iables sep arately. There are several ways to declare a str ing var iable; these are all equivalent: s := ""

var s string

var s = ""

var s string = ""

Why should you prefer one for m to another? The firs t form, a short var iable decl arat ion, is the most compact, but it may be used only wit hin a function, not for package-le vel var iables.

The secon d form relies on defau lt initializat ion to the zero value for str ings, which is "". The third for m is rarely used except when declaring multiple var iables. The fourth for m is explicit ab out the var iable’s typ e, which is redundant when it is the same as that of the initial value but ne cessary in other cas es where the y are not of the same typ e. In prac tice, you should general ly us e on e of the firs t two for ms, wit h explicit initializat ion to say that the initial value is important and imp licit initializat ion to say that the initial value doesn’t matter.

As not ed above , each time aro und the loop, the str ing s gets comp letely new contents. The +=

st atement makes a new str ing by con catenat ing the old str ing , a space charac ter, and the next argument, then assig ns the new str ing to s. The old cont ents of s are no lon g er in use, so the y wi l l be garb age-collec ted in due course.

If the amount of dat a invo l ved is large , this cou ld be costly. A simpler and more efficient solut ion wou ld be to use the Join func tion fro m the strings package:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

8

CHAPTER 1.

TUTORIAL

gopl.io/ch1/echo3

func main() {

fmt.Println(strings.Join(os.Args[1:], " "))

}

Final ly, if we don’t care about for mat but just want to see the values, perhaps for debug ging, we can let Println format the results for us:

fmt.Println(os.Args[1:])

The out put of this statement is like what we wou ld get fro m strings.Join, but wit h sur-ro unding brackets. Any slice may be print ed this way.

Exercis e 1.1: Mo dif y the echo prog ram to als o pr int os.Args[0], the name of the command that invo ked it.

Exercis e 1.2: Mo dif y the echo prog ram to print the index and value of each of its arguments, on e per line.

Exercis e 1.3: Experiment to measure the dif ference in running time bet ween our pot ent ial ly inefficient versions and the one that uses strings.Join. (Section 1.6 illustrates par t of the time package, and Sec tion 11.4 shows how to write benchmark tests for systematic performance evaluation.)

ptg16091132

1.3. Finding Duplicate Lines

Prog rams for file copy ing , pr int ing , searching, sor ting, count ing , and the like all have a simi lar st ruc ture: a loop over the inp ut, som e comp utation on each element, and generat ion of out put on the fly or at the end. We’ll show three var iants of a program cal le d dup; it is par tly ins pired by the Unix uniq command, which looks for adj acent dup lic ate lines. The str uctures and packages used are model s that can be easi ly adapt ed.

The firs t version of dup pr ints each line that app ears more than once in the stand ard inp ut, preceded by its count. This program introduces the if st atement, the map data typ e, and the bufio package.

gopl.io/ch1/dup1

// Dup1 prints the text of each line that appears more than

// once in the standard input, preceded by its count.

package main

import (

"bufio"

"fmt"

"os"

)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 1.3. FINDING DUPLICATE LINES

9

func main() {

counts := make(map[string]int)

input := bufio.NewScanner(os.Stdin)

for input.Scan() {

counts[input.Text()]++

}

// NOTE: ignoring potential errors from input.Err()

for line, n := range counts {

if n > 1 {

fmt.Printf("%d\t%s\n", n, line)

}

}

}

As wit h for, parenthes es are never used aro und the con dit ion in an if st atement, but braces are required for the body. There can be an opt ion al else part that is exe cut ed if the con dit ion is false.

A map holds a set of key/value pairs and provides cons tant-t ime operat ions to store , retr ieve, or test for an item in the set. The key may be of any typ e whos e values can comp are d with ==, st rings being the most common example; the value may be of any typ e at all. In this example, the keys are str ings and the values are ints. The bui lt-in function make creates a new emp ty map; it has other uses too. Maps are dis cussed at lengt h in Sec tion 4.3.

ptg16091132

Each time dup re ads a line of inp ut, the line is used as a key int o the map and the corresp onding value is increment ed. The statement counts[input.Text()]++ is equivalent to thes e two statements:

line := input.Text()

counts[line] = counts[line] + 1

It’s not a pro blem if the map doesn’t yet cont ain that key. The firs t time a new line is seen, the expression counts[line] on the rig ht-hand side evaluates to the zero value for its typ e, which is 0 for int.

To print the results, we use another range-b ased for lo op, this time over the counts map. As before , each iterat ion pro duces two results, a key and the value of the map element for that ke y. The order of map iterat ion is not specified, but in prac tice it is random, varying fro m on e run to another. This desig n is int ent ion al, since it pre vents programs fro m re lying on any particular order ing where non e is guarante e d.

Onward to the bufio package, which helps make inp ut and out put efficient and convenient.

One of its most useful features is a typ e called Scanner that reads inp ut and bre aks it int o lines or words; it’s often the easiest way to pro cess inp ut that com es natural ly in lines.

The program uses a short var iable decl arat ion to cre ate a new var iable input that refers to a bufio.Scanner:

input := bufio.NewScanner(os.Stdin)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

10

CHAPTER 1. TUTORIAL

The scanner reads fro m the program’s stand ard inp ut. Each cal l to input.Scan() re ads the next line and remov es the newline charac ter fro m the end; the result can be ret rie ved by cal ling input.Text(). The Scan func tion retur ns true if there is a line and false when there is no more inp ut.

The function fmt.Printf, like printf in C and other langu ages, pro duces for matte d output from a list of expressions. Its firs t argument is a for mat str ing that specifies how subsequent arguments should be for matte d. The for mat of each argument is deter mined by a conversion ch arac ter, a letter fol low ing a percent sig n. Fo r example, %d formats an integer operand using de cimal not ation, and %s exp ands to the value of a str ing operand.

Printf has over a dozen such conversions, which Go programmers cal l verb s. This table is far from a complete specification but illustrates many of the features that are avai lable:

%d

de cimal integer

%x, %o, %b

integer in hexade cimal, octal, binar y

%f, %g, %e

floating-p oint number: 3.141593 3.141592653589793 3.141593e+00

%t

boole an: true or false

%c

rune (Unico de co de point)

%s

st ring

%q

quot ed str ing "abc" or rune 'c'

%v

any value in a natural for mat

ptg16091132

%T

type of any value

%%

literal percent sig n (no operand)

The for mat str ing in dup1 also contains a tab \t and a newline \n. Str ing lit erals may cont ain such es cap e sequ ences for represent ing other wis e invisible charac ters. Printf do es not write a ne wline by defau lt. By convent ion, for matting functions whose names end in f, such as log.Printf and fmt.Errorf, use the for matting rules of fmt.Printf, where as those whose names end in ln fo llow Println, for matting their arguments as if by %v, fol low ed by a ne wline.

Many programs read eit her fro m their stand ard inp ut, as abov e, or fro m a sequence of named files. The next version of dup can read fro m the stand ard inp ut or handle a list of file names, using os.Open to open each one:

gopl.io/ch1/dup2

// Dup2 prints the count and text of lines that appear more than once

// in the input.

It reads from stdin or from a list of named files.

package main

import (

"bufio"

"fmt"

"os"

)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 1.3. FINDING DUPLICATE LINES

11

func main() {

counts := make(map[string]int)

files := os.Args[1:]

if len(files) == 0 {

countLines(os.Stdin, counts)

} else {

for _, arg := range files {

f, err := os.Open(arg)

if err != nil {

fmt.Fprintf(os.Stderr, "dup2: %v\n", err)

continue

}

countLines(f, counts)

f.Close()

}

}

for line, n := range counts {

if n > 1 {

fmt.Printf("%d\t%s\n", n, line)

}

}

}

func countLines(f *os.File, counts map[string]int) {

input := bufio.NewScanner(f)

ptg16091132

for input.Scan() {

counts[input.Text()]++

}

// NOTE: ignoring potential errors from input.Err()

}

The function os.Open returns two values. The firs t is an open file (*os.File) that is used in su bsequent reads by the Scanner.

The secon d resu lt of os.Open is a value of the bui lt-in error type. If err equals the speci al built-in value nil, the file was opene d successf ully. The file is read, and when the end of the input is reach ed, Close clos es the file and releases any res ources. On the other hand, if err is not nil, som ething went wro ng. In that cas e, the error value descr ibes the pro blem. Our simple-minde d er ror handling prints a message on the stand ard error stream using Fprintf and the verb %v, which displ ays a value of any typ e in a default for mat, and dup then car r ies on with the next file; the continue st atement goes to the next iterat ion of the enclosing for lo op.

In the int erests of keeping code samples to a reasonable size, our early examples are int ent ional ly som ewhat cavalier about error handling. Cle arly we must che ck for an error fro m os.Open; how ever, we are ignor ing the less likely possibi lit y that an erro r could occ ur while re ading the file wit h input.Scan. We will not e pl aces where we’ve skipp ed error che cking , and we will go int o the det ails of error handling in Sec tion 5.4.

No tice that the cal l to countLines precedes its decl arat ion. Functions and other package-le vel entities may be declare d in any order.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

12

CHAPTER 1. TUTORIAL

A map is a reference to the dat a st ruc ture cre ate d by make. When a map is passed to a function, the function receives a copy of the reference, so any changes the cal le d func tion makes to the underly ing dat a st ruc ture will be visible through the cal ler’s map reference too. In our example, the values inserted int o the counts map by countLines are seen by main.

The versions of dup ab ove operate in a ‘‘st reaming’’ mo de in which inp ut is read and bro ken into lines as needed, so in princip le these programs can handle an arbit rar y amount of inp ut.

An alt ernat ive appro ach is to read the ent ire inp ut int o memory in one big gulp, split it int o lines all at once, then pro cess the lines. The fol low ing version, dup3, operates in that fashion.

It int roduces the function ReadFile (f rom the io/ioutil package), which reads the ent ire cont ents of a name d file, and strings.Split, which splits a str ing int o a slice of subst rings.

(Split is the opp osite of strings.Join, which we saw earlier.) We’ve simplified dup3 some what. First, it only reads name d files, not the stand ard inp ut, since ReadFile re quires a file name argument. Secon d, we mov ed the count ing of the lines back into main, since it is now needed in only one place.

gopl.io/ch1/dup3

package main

import (

"fmt"

ptg16091132

"io/ioutil"

"os"

"strings"

)

func main() {

counts := make(map[string]int)

for _, filename := range os.Args[1:] {

data, err := ioutil.ReadFile(filename)

if err != nil {

fmt.Fprintf(os.Stderr, "dup3: %v\n", err)

continue

}

for _, line := range strings.Split(string(data), "\n") {

counts[line]++

}

}

for line, n := range counts {

if n > 1 {

fmt.Printf("%d\t%s\n", n, line)

}

}

}

ReadFile returns a byte slice that must be converted int o a string so it can be split by strings.Split. We will dis cuss str ings and byte slices at lengt h in Sec tion 3.5.4.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 1.4. ANIMATED GIFS

13

Un der the cov ers, bufio.Scanner, ioutil.ReadFile, and ioutil.WriteFile us e the Read and Write methods of *os.File, but it’s rare that most programmers need to access those lower-le vel routines direc tly. The hig her-le vel functions like those fro m bufio and io/ioutil are easier to use.

Exercis e 1.4: Mo dif y dup2 to print the names of all files in which each dup lic ated line occ urs.

1.4. Animated GIFs

The next program demon strates basic usage of Go’s stand ard image packages, which we’ll use to cre ate a sequence of bit-mapped images and then encode the sequence as a GIF animat ion.

The images, cal le d Li ssajous figure s, were a staple visual effec t in sci-fi films of the 1960s. They are the paramet ric cur ves pro duce d by har monic osci l lat ion in two dimensions, such as two sine waves fed int o the x and y inputs of an oscillos cope. Figure 1.1 shows som e examples.

ptg16091132

Figure 1.1. Fo ur Lissajous figures.

There are several new con str ucts in this code, including const de clarat ions, str uct typ es, and comp osite lit erals. Unlike most of our examples, this one als o invo l ves floating-p oint computation s. We’ll dis cuss these topics only briefly here, pushing most det ails off to later chapters, since the primary goal rig ht now is to give you an ide a of what Go looks like and the kind s of things that can be don e easi ly wit h the langu age and its librar ies.

gopl.io/ch1/lissajous

// Lissajous generates GIF animations of random Lissajous figures.

package main

import (

"image"

"image/color"

"image/gif"

"io"

"math"

"math/rand"

"os"

)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

14

CHAPTER 1. TUTORIAL

var palette = []color.Color{color.White, color.Black}

const (

whiteIndex = 0 // first color in palette

blackIndex = 1 // next color in palette

)

func main() {

lissajous(os.Stdout)

}

func lissajous(out io.Writer) {

const (

cycles = 5

// number of complete x oscillator revolutions

res

= 0.001 // angular resolution

size

= 100

// image canvas covers [-size..+size]

nframes = 64

// number of animation frames

delay

= 8

// delay between frames in 10ms units

)

freq := rand.Float64() * 3.0 // relative frequency of y oscillator anim := gif.GIF{LoopCount: nframes}

phase := 0.0 // phase difference

for i := 0; i < nframes; i++ {

rect := image.Rect(0, 0, 2*size+1, 2*size+1)

img := image.NewPaletted(rect, palette)

ptg16091132

for t := 0.0; t < cycles*2*math.Pi; t += res {

x := math.Sin(t)

y := math.Sin(t*freq + phase)

img.SetColorIndex(size+int(x*size+0.5), size+int(y*size+0.5), blackIndex)

}

phase += 0.1

anim.Delay = append(anim.Delay, delay)

anim.Image = append(anim.Image, img)

}

gif.EncodeAll(out, &anim) // NOTE: ignoring encoding errors

}

Af ter importing a package whose pat h has multiple components, like image/color, we refer to the package wit h a name that com es fro m the last component. Thu s the var iable color.White belongs to the image/color package and gif.GIF belongs to image/gif.

A const de clarat ion (§3.6) gives names to con stants, that is, values that are fixe d at comp ile time, such as the numer ical parameters for cyc les, frames, and del ay. Like var de clarat ions, const de clarat ions may appear at package level (so the names are visible throughout the package) or wit hin a function (so the names are visible only wit hin that function). The value of a cons tant must be a number, str ing , or boole an.

The expressions []color.Color{...} and gif.GIF{...} are comp osite lit era ls (§4.2, §4.4.1), a compact not ation for ins tantiat ing any of Go’s composite typ es from a sequence of element values. Here, the firs t on e is a slice and the secon d on e is a st ruc t.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 1.5. FETCHING A URL

15

The typ e gif.GIF is a str uct typ e (§4.4). A st ruc t is a gro up of values cal le d fiel ds, often of different typ es, that are col lec ted toget her in a single obj e ct that can be tre ate d as a unit. The var iable anim is a str uct of typ e gif.GIF. The str uct lit eral creates a str uct value whose LoopCount field is set to nframes; all other fields have the zero value for their typ e. The indiv idu al fields of a str uct can be accessed using dot not ation, as in the final two assig nments which explicitly update the Delay and Image fields of anim.

The lissajous func tion has two neste d lo ops. The out er loop runs for 64 iterat ions, each producing a single frame of the animat ion. It cre ates a new 201&201 image wit h a palette of two color s, white and black. All pixel s are initial ly set to the palette’s zero value (the zerot h color in the palette), which we set to white. Each pass through the inner loop generates a new image by setting som e pixe ls to black. The result is app ended, using the bui lt-in append function (§4.2.1), to a list of frames in anim, along wit h a specified del ay of 80ms. Final ly the sequence of frames and del ays is encoded int o GIF for mat and writt en to the out put stream out. The typ e of out is io.Writer, which lets us write to a wide range of possible destination s, as we’ll show soon.

The inner loop runs the two oscillator s. The x os cillator is just the sine function. The y os cillator is als o a sinus oid, but its fre quency rel ative to the x os cillator is a random number between 0 and 3, and its phase rel ative to the x os cillator is initial ly zero but incre ases wit h each frame of the animat ion. The loop runs unt i l the x os cillator has comp leted five full cycles. At each step, it cal ls SetColorIndex to color the pixel cor responding to (x, y) black, ptg16091132

which is at position 1 in the palette.

The main func tion cal ls the lissajous func tion, direc ting it to write to the stand ard out put, so this command pro duces an animated GIF wit h frames like those in Figure 1.1: $ go build gopl.io/ch1/lissajous

$ ./lissajous >out.gif

Exercis e 1.5: Change the Lissajous program’s color palette to gre en on black, for adde d authenticity. To cre ate the web color #RRGGBB, use color.RGBA{0xRR, 0xGG, 0xBB, 0xff}, where each pair of hexade cimal digits represents the int ensit y of the red, gre en, or blue component of the pixel.

Exercis e 1.6: Mo dif y the Lissajous program to pro duce images in multiple color s by adding more values to palette and then displ aying them by chang ing the third argument of SetColorIndex in som e interest ing way.

1.5. Fet ching a URL

Fo r many app lic ations, access to infor mat ion fro m the Int ernet is as imp ortant as access to the lo cal file system. Go provides a collec tion of packages, gro uped under net, that make it easy to send and receive infor mat ion through the Int ernet, make low-le vel networ k connec tion s, and set up ser vers, for which Go’s con cur rency features (introduce d in Chapt er 8) are par tic ularly useful.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

16

CHAPTER 1. TUTORIAL

To illustrate the minimum necessary to ret rie ve infor mat ion over HTTP, here’s a simple prog ram called fetch that fetch es the content of each specified URL and prints it as uninterpreted text; it’s ins pired by the invaluable utilit y curl. Obv iou sly one wou ld usu ally do more with such dat a, but this shows the basic idea. We will use this program fre quently in the book.

gopl.io/ch1/fetch

// Fetch prints the content found at a URL.

package main

import (

"fmt"

"io/ioutil"

"net/http"

"os"

)

func main() {

for _, url := range os.Args[1:] {

resp, err := http.Get(url)

if err != nil {

fmt.Fprintf(os.Stderr, "fetch: %v\n", err)

os.Exit(1)

}

ptg16091132

b, err := ioutil.ReadAll(resp.Body)

resp.Body.Close()

if err != nil {

fmt.Fprintf(os.Stderr, "fetch: reading %s: %v\n", url, err) os.Exit(1)

}

fmt.Printf("%s", b)

}

}

This program introduces functions fro m two packages, net/http and io/ioutil. The http.Get func tion makes an HTTP request and, if there is no error, retur ns the result in the resp ons e st ruc t resp. The Body field of resp cont ains the ser ver respons e as a readable st ream. Next, ioutil.ReadAll re ads the ent ire resp ons e; the result is store d in b. The Body st ream is clos ed to avoid leak ing res ources, and Printf wr ites the respons e to the stand ard output.

$ go build gopl.io/ch1/fetch

$ ./fetch http://gopl.io

<html>

<head>

<title>The Go Programming Language</title>

...

If the HTTP request fai ls, fetch reports the fai lure ins tead:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 1.6. FETCHING URLS CONCURRENTLY

17

$ ./fetch http://bad.gopl.io

fetch: Get http://bad.gopl.io: dial tcp: lookup bad.gopl.io: no such host In eit her erro r case, os.Exit(1) caus es the pro cess to exit wit h a statu s co de of 1.

Exercis e 1.7: The function cal l io.Copy(dst, src) re ads fro m src and writes to dst. Use it instead of ioutil.ReadAll to copy the respons e body to os.Stdout without requir ing a buffer large enoug h to hold the ent ire stream. Be sure to che ck the error result of io.Copy.

Exercis e 1.8: Mo dif y fetch to add the prefix http:// to each argument URL if it is missing.

Yo u mig ht want to use strings.HasPrefix.

Exercis e 1.9: Mo dif y fetch to als o pr int the HTTP statu s co de, found in resp.Status.

1.6. Fet ching URLs Concurrently

One of the most int erest ing and nove l aspects of Go is its sup por t for con cur rent programming. This is a large topic, to which Chapt er 8 and Chapt er 9 are devot ed, so for now we’ll give you just a taste of Go’s main conc urrency mechanisms, goroutines and channel s.

The next program, fetchall, does the same fetch of a URL’s contents as the pre vious example, but it fetch es many URLs, all con cur rently, so that the pro cess will take no lon g er than the ptg16091132

longest fetch rat her than the sum of all the fetch times. This version of fetchall discards the resp ons es but rep orts the size and elaps ed time for each one: gopl.io/ch1/fetchall

// Fetchall fetches URLs in parallel and reports their times and sizes.

package main

import (

"fmt"

"io"

"io/ioutil"

"net/http"

"os"

"time"

)

func main() {

start := time.Now()

ch := make(chan string)

for _, url := range os.Args[1:] {

go fetch(url, ch) // start a goroutine

}

for range os.Args[1:] {

fmt.Println(<-ch) // receive from channel ch

}

fmt.Printf("%.2fs elapsed\n", time.Since(start).Seconds())

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

18

CHAPTER 1. TUTORIAL

func fetch(url string, ch chan<- string) {

start := time.Now()

resp, err := http.Get(url)

if err != nil {

ch <- fmt.Sprint(err) // send to channel ch

return

}

nbytes, err := io.Copy(ioutil.Discard, resp.Body)

resp.Body.Close() // don't leak resources

if err != nil {

ch <- fmt.Sprintf("while reading %s: %v", url, err) return

}

secs := time.Since(start).Seconds()

ch <- fmt.Sprintf("%.2fs

%7d %s", secs, nbytes, url)

}

Here’s an example:

$ go build gopl.io/ch1/fetchall

$ ./fetchall https://golang.org http://gopl.io https://godoc.org 0.14s

6852

https://godoc.org

0.16s

7261

https://golang.org

0.48s

2475

http://gopl.io

ptg16091132

0.48s elapsed

A goro utine is a con cur rent function exe cut ion. A ch ann el is a communic ation mechanism that allows one goroutine to pass values of a specified typ e to another goroutine. The function main runs in a goroutine and the go st atement cre ates addition al goroutines.

The main func tion cre ates a channel of str ings using make. For each command-line argument, the go st atement in the firs t range loop starts a new goroutine that cal ls fetch asy nchro nou sly to fetch the URL using http.Get. The io.Copy func tion reads the body of the respons e and discards it by writing to the ioutil.Discard output stream. Copy returns the byte count, along wit h any error that occ urre d. As each result arrives, fetch send s a summar y line on the ch annel ch. The secon d range loop in main re ceives and prints those lines.

Wh en one goroutine att emp ts a send or receive on a channel, it blo cks until another goroutine attemp ts the cor responding receive or send operat ion, at which point the value is transfer red and bot h goro utines pro ceed. In this example, each fetch send s a value (ch <- expre ssi on) on the channel ch, and main re ceives all of them (<-ch). Hav ing main do all the print ing ens ures that out put fro m each goroutine is pro cessed as a unit, wit h no danger of int erleaving if two goro utines finish at the same time.

Exercis e 1.10: Find a web sit e that pro duces a large amount of dat a. Invest igate caching by running fetchall twice in succession to see whether the rep orted time changes much. Do you get the same content each time? Modif y fetchall to print its out put to a file so it can be examined.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 1.7. A WEB SERVER

19

Exercis e 1.11: Tr y fetchall with lon g er argument lists, such as samples fro m the top million we b sites avai lable at alexa.com. How does the program beh ave if a we b site just doesn’t resp ond? (Section 8.9 descr ibes mechanisms for coping in such cas es.) 1.7. A Web Server

Go’s librar ies makes it easy to write a web ser ver that respond s to client requests like those made by fetch. In this sec tion, we’ll show a minimal ser ver that retur ns the pat h comp onent of the URL used to access the ser ver. That is, if the request is for http://local-

host:8000/hello, the respons e wi l l be URL.Path = "/hello".

gopl.io/ch1/server1

// Server1 is a minimal "echo" server.

package main

import (

"fmt"

"log"

"net/http"

)

func main() {

ptg16091132

http.HandleFunc("/", handler) // each request calls handler log.Fatal(http.ListenAndServe("localhost:8000", nil))

}

// handler echoes the Path component of the request URL r.

func handler(w http.ResponseWriter, r *http.Request) {

fmt.Fprintf(w, "URL.Path = %q\n", r.URL.Path)

}

The program is only a handf ul of lines long because librar y func tions do most of the wor k.

The main func tion conne cts a handler function to incoming URLs that beg in with /, which is al l URLs, and starts a ser ver listening for incoming requests on por t 8000. A re quest is represent ed as a str uct of typ e http.Request, which contains a number of rel ate d fields, one of which is the URL of the incoming request. When a request arrives, it is given to the handler func tion, which ext racts the pat h comp onent (/hello) fro m the request URL and sends it back as the respons e, using fmt.Fprintf. Web ser vers will be explained in det ail in Section 7.7.

Let’s start the ser ver in the backg round. On Mac OS X or Linux, add an ampers and (&) to the command; on Micros oft Windows, you will need to run the command wit hout the amp ers and in a sep arate command window.

$ go run src/gopl.io/ch1/server1/main.go &

We can then make client requests fro m the command line:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

20

CHAPTER 1. TUTORIAL

$ go build gopl.io/ch1/fetch

$ ./fetch http://localhost:8000

URL.Path = "/"

$ ./fetch http://localhost:8000/help

URL.Path = "/help"

Alternat ive ly, we can access the ser ver fro m a web brows er, as shown in Figure 1.2.

Figure 1.2. A respons e from the ech o server.

It’s easy to add features to the ser ver. One useful addition is a specific URL that retur ns a st atu s of som e sort. For example, this version does the same ech o but als o counts the number of requests; a request to the URL /count returns the count so far, exc luding /count re quests themselves:

ptg16091132

gopl.io/ch1/server2

// Server2 is a minimal "echo" and counter server.

package main

import (

"fmt"

"log"

"net/http"

"sync"

)

var mu sync.Mutex

var count int

func main() {

http.HandleFunc("/", handler)

http.HandleFunc("/count", counter)

log.Fatal(http.ListenAndServe("localhost:8000", nil))

}

// handler echoes the Path component of the requested URL.

func handler(w http.ResponseWriter, r *http.Request) {

mu.Lock()

count++

mu.Unlock()

fmt.Fprintf(w, "URL.Path = %q\n", r.URL.Path)

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 1.7. A WEB SERVER

21

// counter echoes the number of calls so far.

func counter(w http.ResponseWriter, r *http.Request) {

mu.Lock()

fmt.Fprintf(w, "Count %d\n", count)

mu.Unlock()

}

The ser ver has two handlers, and the request URL deter mines which one is cal le d: a request for /count invo kes counter and all others inv oke handler. A hand ler pattern that ends wit h a slash match es any URL that has the pattern as a prefix. Behind the scenes, the ser ver runs the handler for each incoming request in a sep arate goroutine so that it can ser ve multiple re quests simultane ously. How ever, if two con cur rent requests try to update count at the same time, it mig ht not be increment ed con sistent ly; the program wou ld have a ser ious bug cal le d a ra c e condit ion (§9.1). To avoid this pro blem, we must ens ure that at most one goroutine accesses the var iable at a time, which is the pur pos e of the mu.Lock() and mu.Unlock() calls that bracket each access of count. We’ll look more clos ely at con cur rency wit h sh are d var iables in Chapt er 9.

As a richer example, the handler function can rep ort on the headers and for m data that it re ceives, mak ing the ser ver useful for ins pec ting and debug ging requests: gopl.io/ch1/server3

// handler echoes the HTTP request.

ptg16091132

func handler(w http.ResponseWriter, r *http.Request) {

fmt.Fprintf(w, "%s %s %s\n", r.Method, r.URL, r.Proto) for k, v := range r.Header {

fmt.Fprintf(w, "Header[%q] = %q\n", k, v)

}

fmt.Fprintf(w, "Host = %q\n", r.Host)

fmt.Fprintf(w, "RemoteAddr = %q\n", r.RemoteAddr) if err := r.ParseForm(); err != nil {

log.Print(err)

}

for k, v := range r.Form {

fmt.Fprintf(w, "Form[%q] = %q\n", k, v)

}

}

This uses the fields of the http.Request st ruc t to pro duce out put like this: GET /?q=query HTTP/1.1

Header["Accept-Encoding"] = ["gzip, deflate, sdch"]

Header["Accept-Language"] = ["en-US,en;q=0.8"]

Header["Connection"] = ["keep-alive"]

Header["Accept"] = ["text/html,application/xhtml+xml,application/xml;..."]

Header["User-Agent"] = ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5)..."]

Host = "localhost:8000"

RemoteAddr = "127.0.0.1:59911"

Form["q"] = ["query"]

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

22

CHAPTER 1. TUTORIAL

No tice how the cal l to ParseForm is neste d within an if st atement. Go allows a simple statement such as a local variable declarat ion to pre cede the if condit ion, which is par tic ularly us eful for error handling as in this example. We cou ld have writt en it as err := r.ParseForm()

if err != nil {

log.Print(err)

}

but com bining the statements is shorter and reduces the scope of the var iable err, which is go o d prac tice. We’ll define scope in Sec tion 2.7.

In these programs, we’ve seen three ver y dif ferent typ es us ed as out put streams. The fetch prog ram copied HTTP respons e data to os.Stdout, a file, as did the lissajous prog ram.

The fetchall prog ram thre w the respons e away (whi le counting its lengt h) by copying it to the trivial sin k ioutil.Discard. And the web ser ver abov e us ed fmt.Fprintf to write to an http.ResponseWriter repres enting the web brows er.

Although these three typ es dif fer in the det ails of what the y do, the y al l satisf y a common interface, allow ing any of them to be used where ver an out put stream is needed. That int erface, cal le d io.Writer, is dis cussed in Sec tion 7.1.

Go’s int erface mechanism is the topic of Chapt er 7, but to give an ide a of what it’s cap able of, ptg16091132

let’s see how easy it is to com bine the web ser ver wit h the lissajous func tion so that animate d GIFs are writt en not to the stand ard out put, but to the HTTP client. Just add these lines to the web ser ver :

handler := func(w http.ResponseWriter, r *http.Request) {

lissajous(w)

}

http.HandleFunc("/", handler)

or equivalently:

http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {

lissajous(w)

})

The secon d argument to the HandleFunc func tion cal l immediate ly above is a func tion lit era l, that is, an anonymou s func tion define d at its point of use. We will explain it fur ther in Section 5.6.

Once you’ve made this change , visit http://localhost:8000 in your brows er. Each time you lo ad the page, you’ll see a new animat ion like the one in Figure 1.3.

Exercis e 1.12: Mo dif y the Lissajous ser ver to read parameter values fro m the URL. For example, you mig ht arrange it so that a URL like http://localhost:8000/?cycles=20 sets the numb er of cyc les to 20 ins tead of the defau lt 5. Use the strconv.Atoi func tion to convert the st ring parameter into an int eger. You can see its document ation wit h go doc strconv.Atoi.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 1.8. LOOSE ENDS

23

Figure 1.3. Animated Lissajous figures in a brows er.

1.8. Loose Ends

ptg16091132

There is a lot more to Go than we’ve cov ere d in this quick int roduc tion. Here are som e topics we’ve barely touch ed upon or omitt ed ent ire ly, wit h ju st enoug h discussion that the y wi l l be fami liar when the y make brief appearances before the full treatment.

Contro l flow : We cov ere d the two fundament al control-flow statements, if and for, but not the switch st atement, which is a multi-way branch. Here’s a small example: switch coinflip() {

case "heads":

heads++

case "tails":

tails++

default:

fmt.Println("landed on edge!")

}

The result of cal ling coinflip is compare d to the value of each cas e. Cas es are evaluate d from top to bot tom, so the firs t matching one is exe cut ed. The opt ion al defau lt cas e match es if non e of the other cas es do es; it may be place d anywhere . Cases do not fal l thro ugh fro m on e to the next as in C-like langu ages (thoug h there is a rarely used fallthrough st atement that overrides this beh avior).

A switch do es not need an operand; it can just list the cas es, each of which is a boole an expression:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

24

CHAPTER 1. TUTORIAL

func Signum(x int) int {

switch {

case x > 0:

return +1

default:

return 0

case x < 0:

return -1

}

}

This for m is cal le d a tagless switch; it’s equivalent to switch true.

Like the for and if st atements, a switch may include an opt ion al simple statement—a short var iable decl arat ion, an increment or assig nment statement, or a function cal l—that can be us ed to set a value before it is teste d.

The break and continue st atements modif y the flow of control . A break caus es control to resume at the next statement after the inner most for, switch, or select st atement (which we’ll see later), and as we saw in Sec tion 1.3, a continue caus es the inner most for lo op to st art its next iterat ion. Statements may be lab ele d so that break and continue can refer to them, for ins tance to bre ak out of several neste d lo ops at once or to start the next iterat ion of the out ermost loop. There is even a goto st atement, thoug h it’s int ended for machine-gener-ptg16091132

ated code, not regu lar use by programmers.

Na med types: A type de clarat ion makes it possible to give a name to an exist ing typ e. Since st ruc t types are often long , they are nearly always named. A fami liar example is the definition of a Point type for a 2-D graphics system:

type Point struct {

X, Y int

}

var p Point

Type declarat ions and named typ es are cov ere d in Chapt er 2.

Po int ers: Go provides point ers, that is, values that contain the address of a var iable. In som e languages, not ably C, point ers are relative ly uncon strained. In other langu ages, point ers are disguised as ‘‘references,’’ and there’s not much that can be don e with them except pass them around. Go takes a position som ewhere in the midd le. Point ers are explicitly visible. The & op erator yields the address of a var iable, and the * op erator ret rie ves the var iable that the pointer refers to, but there is no point er arithmetic. We’ll explain point ers in Sec tion 2.3.2.

Me thods and inte rfa ces: A met hod is a function ass oci ated wit h a named typ e; Go is unu sual in that met hods may be att ach ed to almost any named typ e. Met hods are cov ere d in Chapter 6. Int erfaces are abstrac t types that let us tre at dif ferent con crete typ es in the same way based on what met hods the y have , not how the y are represent ed or imp lemente d. Interfaces are the subjec t of Chapt er 7.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 1.8. LOOSE ENDS

25

Pa ckage s: Go com es wit h an extensive stand ard librar y of useful packages, and the Go community has create d and share d many more. Programming is often more about using exist ing packages than about writing original code of one’s own. Throughout the book, we will point out a couple of dozen of the most imp ortant stand ard packages, but there are many more we don’t have space to mention, and we cannot provide anyt hing remot ely like a complete reference for any package.

Before you emb ark on any new program, it’s a good ide a to see if packages already exist that mig ht help you get your job don e more easi ly. You can find an index of the stand ard librar y packages at https://golang.org/pkg and the packages cont ribut ed by the community at

https://godoc.org. The go doc to ol makes these documents easi ly accessible fro m the command line:

$ go doc http.ListenAndServe

package http // import "net/http"

func ListenAndServe(addr string, handler Handler) error ListenAndServe listens on the TCP network address addr and then calls Serve with handler to handle requests on incoming connections.

...

Comm ents: We have already mention ed documentation comments at the beg inning of a prog ram or package. It’s als o go o d st yle to write a comment before the declarat ion of each func tion to specif y its beh avior. These convent ion s are imp ortant, because the y are used by ptg16091132

to ols like go doc and godoc to locate and displ ay documentation (§10.7.4).

Fo r comments that span multiple lines or app ear wit hin an expression or statement, there is also the /* ... */ notation fami liar fro m ot her langu ages. Such comments are som etimes us ed at the beg inning of a file for a large blo ck of explanatory text to avoid a // on every line.

Wi thin a comment, // and /* have no speci al me aning, so comments do not nest.

This page intentionally left blank

ptg16091132

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

2

Program Structure

In Go, as in any other programming langu age, one bui lds large programs fro m a small set of basic cons tructs. Var iables store values. Simple expressions are com bine d into larger ones with operat ions like addition and subtrac tion. Basic typ es are col lec ted int o ag gregates like ar rays and str ucts. Expressions are used in statements whose exe cut ion order is deter mined ptg16091132

by control-flow statements like if and for. Statements are gro uped int o func tions for is olation and reuse. Functions are gat hered int o source files and packages.

We saw examples of most of these in the pre vious chapt er. In this chapt er, we’ll go int o more det ail about the basic str uctural elements of a Go program. The example programs are int ention ally simple, so we can focus on the langu age wit hout getting sidet racke d by complic ated algor it hms or dat a st ruc tures.

2.1. Names

The names of Go functions, var iables, cons tants, typ es, st atement lab els, and packages fol low a simple rule: a name beg ins wit h a letter (that is, anyt hing that Unico de de ems a letter) or an underscore and may have any number of addition al letters, dig its, and underscores. Cas e matters: heapSort and Heapsort are dif ferent names.

Go has 25 ke yword s li ke if and switch that may be used only where the syntax per mits; the y can’t be used as names.

break

default

func

interface

select

case

defer

go

map

struct

chan

else

goto

package

switch

const

fallthrough

if

range

type

continue

for

import

return

var

27

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

28

CHAPTER 2. PROGRAM STRUCTURE

In addition, there are about three dozen pred eclare d names like int and true for bui lt-in const ants, typ es, and functions:

Cons tants:

true false iota nil

Types:

int int8 int16 int32 int64

uint uint8 uint16 uint32 uint64 uintptr

float32 float64 complex128 complex64

bool byte rune string error

Func tions:

make len cap new append copy close delete

complex real imag

panic recover

Thes e names are not res erve d, so you may use them in declarat ions. We’ll see a handf ul of pl aces where rede claring one of them makes sense, but beware of the pot ent ial for confusion.

If an ent ity is declare d within a function, it is local to that function. If declare d outside of a func tion, how ever, it is visible in all files of the package to which it belon gs. The cas e of the firs t letter of a name deter mines its visibilit y across package bound aries. If the name beg ins with an upp er-case letter, it is expor ted, which means that it is visible and accessible outside of its own package and may be refer red to by other par ts of the program, as wit h Printf in the fmt package. Package names thems elves are always in lower cas e.

ptg16091132

There is no limit on name lengt h, but convent ion and sty le in Go programs lean toward short names, especi ally for local variables wit h smal l scop es; you are much more likely to see var iables named i than theLoopIndex. General ly, the larger the scope of a name, the lon g er and more meaningf ul it should be.

St ylist ically, Go programmers use ‘‘camel cas e’’ when for ming names by com bining words; that is, int erior capit al letters are preferred over int erior underscores. Thus the stand ard librar ies have functions wit h names like QuoteRuneToASCII and parseRequestLine but never quote_rune_to_ASCII or parse_request_line. The letters of acronyms and initialisms like ASCII and HTML are always rendered in the same cas e, so a function mig ht be cal le d html-Escape, HTMLEscape, or escapeHTML, but not escapeHtml.

2.2. Declarations

A decl arati on names a program entity and specifies som e or all of its pro per ties. There are four maj or kinds of declarat ions: var, const, type, and func. We’ll tal k ab out var iables and types in this chapt er, con stants in Chapt er 3, and functions in Chapt er 5.

A Go program is store d in one or more files whose names end in .go. Each file beg ins wit h a package de clarat ion that says what package the file is par t of . The package de clarat ion is fo llowe d by any import de clarat ions, and then a sequence of pack age-level de clarat ions of types, variables, cons tants, and functions, in any order. For example, this program decl ares a cons tant, a function, and a couple of var iables:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 2.2. DECLARATIONS

29

gopl.io/ch2/boiling

// Boiling prints the boiling point of water.

package main

import "fmt"

const boilingF = 212.0

func main() {

var f = boilingF

var c = (f - 32) * 5 / 9

fmt.Printf("boiling point = %g°F or %g°C\n", f, c)

// Output:

// boiling point = 212°F or 100°C

}

The con stant boilingF is a package-le vel declarat ion (as is main), where as the var iables f and c are local to the function main. The name of each package-le vel ent ity is visible not only thro ughout the source file that contains its decl arat ion, but throughout all the files of the package. By contrast, local declarat ions are visible only wit hin the function in which the y are de clare d and perhaps only wit hin a smal l part of it.

A function declarat ion has a name, a list of parameters (the var iables whose values are prov ide d by the function’s cal lers), an opt ion al list of results, and the function body, which ptg16091132

cont ains the statements that define what the function does. The result list is omitt ed if the func tion does not retur n anyt hing. Exe cut ion of the function beg ins wit h the firs t st atement and continues until it encount ers a retur n st atement or reach es the end of a function that has no results. Control and any results are then retur ned to the cal ler.

We’ve seen a fair number of functions already and there are lots more to com e, including an extensive dis cussion in Chapt er 5, so this is only a sketch. The function fToC below enc apsulates the temperature conversion log ic so that it is define d on ly once but may be used fro m mu ltiple places. Here main calls it twice, using the values of two dif ferent local cons tants: gopl.io/ch2/ftoc

// Ftoc prints two Fahrenheit-to-Celsius conversions.

package main

import "fmt"

func main() {

const freezingF, boilingF = 32.0, 212.0

fmt.Printf("%g°F = %g°C\n", freezingF, fToC(freezingF)) // "32°F = 0°C"

fmt.Printf("%g°F = %g°C\n", boilingF, fToC(boilingF))

// "212°F = 100°C"

}

func fToC(f float64) float64 {

return (f - 32) * 5 / 9

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

30

CHAPTER 2. PROGRAM STRUCTURE

2.3. Variables

A var de clarat ion cre ates a var iable of a par tic ular typ e, att ach es a name to it, and sets its initial value. Each declarat ion has the general for m

var name type = expression

Either the typ e or the = expression part may be omitt ed, but not bot h. If the typ e is omitt ed, it is deter mined by the initializer expression. If the expression is omitt ed, the initial value is the zero value for the typ e, which is 0 for numbers, false for boole ans, "" for str ings, and nil for int erfaces and reference typ es (slice, point er, map, channel, function). The zero value of an ag gregate typ e li ke an array or a str uct has the zero value of all of its elements or fields.

The zero-value mechanism ensures that a var iable always holds a wel l-define d value of its typ e; in Go there is no such thing as an uninitialize d var iable. This simplifies code and often ensures sensible beh avior of bound ary con dit ion s without ext ra work. For example, var s string

fmt.Println(s) // ""

pr ints an emp ty str ing , rat her than causing som e kind of error or unp redic table beh avior. Go prog rammers often go to som e ef for t to make the zero value of a more complic ated typ e me aningf ul, so that var iables beg in life in a useful state.

ptg16091132

It is possible to declare and opt ion ally initialize a set of var iables in a single declarat ion, wit h a matching list of expressions. Omitt ing the typ e al lows decl arat ion of multiple var iables of different typ es:

var i, j, k int

// int, int, int

var b, f, s = true, 2.3, "four" // bool, float64, string Init ializers may be lit eral values or arbit rar y expressions. Package-le vel var iables are initialize d before main begins (§2.6.2), and local variables are initialize d as their decl arat ions are encountere d during function exe cut ion.

A set of var iables can also be initialize d by cal ling a function that retur ns multiple values: var f, err = os.Open(name) // os.Open returns a file and an error 2.3.1. Short Variable Declarations

Wi thin a function, an alternate for m called a sh ort var iab le declarati on may be used to declare and initialize local var iables. It takes the for m name := expression, and the typ e of name is deter mined by the typ e of expression. Here are three of the many short var iable decl arat ions in the lissajous func tion (§1.4):

anim := gif.GIF{LoopCount: nframes}

freq := rand.Float64() * 3.0

t := 0.0

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 2.3. VARIABLES

31

Becaus e of their bre vit y and flexibi lit y, short var iable decl arat ions are used to declare and initialize the maj ority of local variables. A var de clarat ion tends to be res erve d for local variables that need an explicit typ e that differs fro m that of the initializer expression, or for when the var iable will be assig ned a value later and its initial value is unimportant.

i := 100

// an int

var boiling float64 = 100 // a float64

var names []string

var err error

var p Point

As wit h var de clarat ions, multiple var iables may be declare d and initialize d in the same short var iable decl arat ion,

i, j := 0, 1

but declarat ions wit h mu ltiple initializer expressions should be used only when the y help read-abilit y, such as for short and natural gro upings like the initializat ion par t of a for lo op.

Ke ep in mind that := is a declarat ion, where as = is an assig nment. A mu lti-var iable decl aration should not be confused wit h a tupl e assig nment (§2.4.1), in which each var iable on the lef t-hand side is assig ned the cor responding value fro m the rig ht-hand side: i, j = j, i // swap values of i and j

ptg16091132

Like ordinar y var de clarat ions, short var iable decl arat ions may be used for cal ls to functions li ke os.Open that retur n two or more values:

f, err := os.Open(name)

if err != nil {

return err

}

// ...use f...

f.Close()

One subtle but imp ortant point: a short var iable decl arat ion does not necessarily decl are al l the var iables on its left-hand side. If som e of them were already declare d in the same lexic al block (§2.7), then the short var iable decl arat ion acts like an assig nment to those var iables.

In the code below, the firs t st atement declares bot h in and err. The secon d de clares out but on ly assig ns a value to the exist ing err var iable.

in, err := os.Open(infile)

// ...

out, err := os.Create(outfile)

A short var iable decl arat ion must declare at least one new var iable, how ever, so this code will not compi le:

f, err := os.Open(infile)

// ...

f, err := os.Create(outfile) // compile error: no new variables

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

32

CHAPTER 2. PROGRAM STRUCTURE

The fix is to use an ordinar y assig nment for the secon d st atement.

A short var iable decl arat ion acts like an assig nment only to var iables that were already de clare d in the same lexic al block; decl arat ions in an out er blo ck are ignored. We’ll see examples of this at the end of the chapt er.

2.3.2. Pointers

A variab le is a pie ce of storage containing a value. Var iables create d by declarat ions are identified by a name, such as x, but many var iables are identified only by expressions like x[i] or x.f. All these expressions read the value of a var iable, except when the y appear on the lef thand side of an assig nment, in which cas e a new value is assig ned to the var iable.

A pointer value is the ad dre ss of a var iable. A pointer is thu s the locat ion at which a value is stored. Not every value has an address, but every var iable does. Wit h a point er, we can read or update the value of a var iable in direc tly, wit hout using or even knowing the name of the var iable, if indeed it has a name.

If a var iable is declare d var x int, the expression &x (‘‘address of x’’) yields a point er to an integer var iable, that is, a value of typ e *int, which is pro nounced ‘‘pointer to int.’’ If this value is cal le d p, we say ‘‘p points to x,’’ or equivalently ‘‘p cont ains the address of x.’’ The var i-ptg16091132

able to which p points is writt en *p. The expression *p yields the value of that var iable, an int, but since *p denot es a var iable, it may also app ear on the lef t-hand side of an assig nment, in which cas e the assig nment updates the var iable.

x := 1

p := &x

// p, of type *int, points to x

fmt.Println(*p) // "1"

*p = 2

// equivalent to x = 2

fmt.Println(x) // "2"

Each component of a var iable of aggregate typ e—a field of a str uct or an element of an array—

is als o a var iable and thu s has an address too.

Variables are som etimes des crib ed as ad dre ssabl e values. Expressions that denot e var iables are the only expressions to which the ad dre ss-of op erator & may be app lie d.

The zero value for a point er of any typ e is nil. The test p != nil is true if p points to a var iable. Point ers are comparable; two point ers are equ al if and only if the y point to the same var iable or bot h are nil.

var x, y int

fmt.Println(&x == &x, &x == &y, &x == nil) // "true false false"

It is per fec tly safe for a function to retur n the address of a local variable. For ins tance, in the co de below, the local variable v create d by this par tic ular cal l to f wi l l remain in existence even af ter the cal l has retur ned, and the point er p wi l l st i l l refer to it:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 2.3. VARIABLES

33

var p = f()

func f() *int {

v := 1

return &v

}

Each cal l of f returns a distinc t value:

fmt.Println(f() == f()) // "false"

Becaus e a point er contains the address of a var iable, passing a point er argument to a function makes it possible for the function to update the var iable that was indirec tly passed. For example, this function increments the var iable that its argument points to and retur ns the new value of the var iable so it may be used in an expression: func incr(p *int) int {

*p++ // increments what p points to; does not change p

return *p

}

v := 1

incr(&v)

// side effect: v is now 2

fmt.Println(incr(&v)) // "3" (and v is 3) Each time we take the address of a var iable or copy a point er, we cre ate new aliases or ways to ptg16091132

identify the same var iable. For example, *p is an ali as for v. Point er ali asing is useful because it allows us to access a var iable wit hout using its name, but this is a dou ble-e dged sword: to find all the statements that access a var iable, we have to know all its alias es. It’s not just pointers that cre ate ali ases; aliasing als o occurs when we copy values of other reference typ es li ke slices, maps, and channel s, and even str ucts, arrays, and int erfaces that contain these typ es.

Po int ers are key to the flag package, which uses a program’s command-line arguments to set the values of cer tain variables dist ribut ed throughout the program. To illustrate, this var iat ion on the earlier echo command takes two opt ion al flags: -n caus es echo to omit the trai ling ne wline that wou ld nor mal ly be print ed, and -s sep caus es it to sep arate the out put arguments by the contents of the str ing sep instead of the defau lt single space. Since this is our four th version, the package is cal le d gopl.io/ch2/echo4.

gopl.io/ch2/echo4

// Echo4 prints its command-line arguments.

package main

import (

"flag"

"fmt"

"strings"

)

var n = flag.Bool("n", false, "omit trailing newline") var sep = flag.String("s", " ", "separator")

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

34

CHAPTER 2. PROGRAM STRUCTURE

func main() {

flag.Parse()

fmt.Print(strings.Join(flag.Args(), *sep))

if !*n {

fmt.Println()

}

}

The function flag.Bool creates a new flag variable of typ e bool. It takes three arguments: the name of the flag ("n"), the var iable’s defau lt value (false), and a message that will be print ed if the user provides an invalid argument, an invalid flag , or -h or -help. Simi larly, flag.String takes a name, a defau lt value, and a mess age , and cre ates a string var iable. The var iables sep and n are point ers to the flag variables, which must be accessed indirec tly as

*sep and *n.

Wh en the program is run, it must cal l flag.Parse before the flags are used, to update the flag var iables fro m their default values. The non-flag arguments are avai lable fro m flag.Args() as a slice of str ings. If flag.Parse encounters an error, it prints a usage message and cal ls os.Exit(2) to terminate the program.

Let’s run som e test cas es on echo:

$ go build gopl.io/ch2/echo4

$ ./echo4 a bc def

ptg16091132

a bc def

$ ./echo4 -s / a bc def

a/bc/def

$ ./echo4 -n a bc def

a bc def$

$ ./echo4 -help

Usage of ./echo4:

-n

omit trailing newline

-s string

separator (default " ")

2.3.3. The new Function

Anot her way to cre ate a var iable is to use the bui lt-in function new. The expression new(T) creates an unname d variab le of typ e T, initializes it to the zero value of T, and retur ns its address, which is a value of typ e *T.

p := new(int)

// p, of type *int, points to an unnamed int variable

fmt.Println(*p) // "0"

*p = 2

// sets the unnamed int to 2

fmt.Println(*p) // "2"

A var iable create d with new is no dif ferent fro m an ordinar y lo cal variable whose address is taken, except that there’s no need to inv ent (and declare) a dummy name, and we can use new(T) in an expression. Thus new is only a syntactic conv enience, not a fundament al not ion:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 2.3. VARIABLES

35

the two newInt func tions below have identical beh avior s.

func newInt() *int {

func newInt() *int {

return new(int)

var dummy int

}

return &dummy

}

Each cal l to new returns a distinc t var iable wit h a unique address: p := new(int)

q := new(int)

fmt.Println(p == q) // "false"

There is one exception to this rule: two var iables whose typ e carries no infor mat ion and is therefore of size zero, such as struct{} or [0]int, may, dep ending on the imp lementation, have the same address.

The new func tion is rel ative ly rarely used because the most common unnamed var iables are of st ruc t types, for which the str uct lit eral synt ax (§4.4.1) is more flexible.

Since new is a pre declare d func tion, not a key word, it’s possible to redefine the name for something els e within a function, for example:

func delta(old, new int) int { return new - old }

Of course, wit hin delta, the bui lt-in new func tion is unavai lable.

ptg16091132

2.3.4. Lifetime of Variables

The li fetim e of a var iable is the int erval of time dur ing which it exists as the program exec utes.

The lifet ime of a package-le vel var iable is the ent ire exe cut ion of the program. By contrast, lo cal variables have dynamic lifet imes: a new ins tance is cre ate d each time the declarat ion st atement is exe cut ed, and the var iable lives on unt i l it becom es unre ach abl e, at which point its storage may be rec ycled. Function parameters and results are local variables too; the y are created each time their enclosing function is cal le d.

Fo r example, in this excer pt fro m the Lissajous program of Sec tion 1.4, for t := 0.0; t < cycles*2*math.Pi; t += res {

x := math.Sin(t)

y := math.Sin(t*freq + phase)

img.SetColorIndex(size+int(x*size+0.5), size+int(y*size+0.5), blackIndex)

}

the var iable t is cre ate d each time the for lo op beg ins, and new var iables x and y are cre ate d on each iterat ion of the loop.

Ho w do es the garb age col lec tor know that a var iable’s storage can be reclaimed? The full story is much more det aile d than we need here, but the basic idea is that every package-le vel var iable, and every local variable of each cur rently active function, can pot ent ial ly be the start or

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

36

CHAPTER 2. PROGRAM STRUCTURE

ro ot of a pat h to the var iable in question, fol low ing point ers and other kinds of references that ultimate ly lead to the var iable. If no such pat h exists, the var iable has becom e unreach able, so it can no lon g er affec t the rest of the computation.

Becaus e the lifet ime of a var iable is deter mined only by whether or not it is reach able, a local var iable may out live a single iterat ion of the enclosing loop. It may cont inue to exist even after its enclosing function has retur ned.

A compi ler may cho ose to allocate local variables on the heap or on the stack but, perhaps surpr isingly, this choice is not deter mined by whether var or new was used to declare the var iable.

var global *int

func f() {

func g() {

var x int

y := new(int)

x = 1

*y = 1

global = &x

}

}

Here , x mu st be heap-al locate d becaus e it is still reach able fro m the var iable global af ter f has retur ned, despit e being declare d as a local var iable; we say x es cap es fro m f. Convers ely, when g returns, the var iable *y becomes unreach able and can be rec ycled. Since *y do es not es cap e from g, it’s safe for the compi ler to allocate *y on the stack, even thoug h it was allocate d with new. In any cas e, the notion of escaping is not som ething that you need to wor ry ptg16091132

ab out in order to write cor rec t co de, thoug h it’s good to keep in mind dur ing per for mance opt imizat ion, since each var iable that escap es re quires an extra memor y al location.

Garb age col lec tion is a tremendou s help in writing cor rec t prog rams, but it does not relie ve you of the burden of thin king about memor y. You don’t need to explicitly allocate and fre e memory, but to write efficient programs you still need to be aware of the lifet ime of var iables.

Fo r example, keeping unnecessary point ers to short-live d objec ts wit hin long-live d objec ts, especi ally global var iables, will pre vent the garb age col lec tor fro m re claiming the short-live d objec ts.

2.4. Assignments

The value held by a var iable is updated by an assig nment statement, which in its simplest for m has a var iable on the lef t of the = sig n and an expression on the rig ht.

x = 1

// named variable

*p = true

// indirect variable

person.name = "bob"

// struct field

count[x] = count[x] * scale // array or slice or map element Each of the arithmetic and bit w ise binar y op erator s has a cor responding assig nment operator al lowing , for example, the last statement to be rewritt en as count[x] *= scale

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 2.4. ASSIGNMENTS

37

which saves us fro m having to rep eat (and re-e valuate) the expression for the var iable.

Numeric var iables can also be increment ed and decrement ed by ++ and -- st atements: v := 1

v++

// same as v = v + 1; v becomes 2

v--

// same as v = v - 1; v becomes 1 again

2.4.1. Tuple Assignment

Anot her for m of assig nment, known as tupl e assig nment, allows several variables to be assig ned at once. All of the rig ht-hand side expressions are evaluate d before any of the var iables are updated, mak ing this for m most useful when som e of the var iables appear on bot h sides of the assig nment, as happens, for example, when swapping the values of two var iables: x, y = y, x

a[i], a[j] = a[j], a[i]

or when computing the gre atest common div isor (GCD) of two int egers: func gcd(x, y int) int {

for y != 0 {

x, y = y, x%y

}

ptg16091132

return x

}

or when computing the n-t h Fibonacci number iterat ive ly: func fib(n int) int {

x, y := 0, 1

for i := 0; i < n; i++ {

x, y = y, x+y

}

return x

}

Tu ple assig nment can also make a sequence of trivial assig nments more compact, i, j, k = 2, 3, 5

though as a matter of sty le, avoid the tup le for m if the expressions are complex; a sequence of separate statements is easier to read.

Certain expressions, such as a cal l to a function wit h mu ltiple results, pro duce several values.

Wh en such a cal l is used in an assig nment statement, the lef t-hand side must have as many var iables as the function has results.

f, err = os.Open("foo.txt")

// function call returns two values

Of ten, functions use these addition al results to indic ate som e kind of error, eit her by retur ning an error as in the cal l to os.Open, or a bool, usu ally cal le d ok. As we’ll see in later chapt ers,

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

38

CHAPTER 2. PROGRAM STRUCTURE

there are three operator s that som etimes beh ave this way too. If a map lookup (§4.3), typ e assertion (§7.10), or channel receive (§8.4.2) appears in an assig nment in which two results are exp ected, each pro duces an addition al boole an resu lt: v, ok = m[key]

// map lookup

v, ok = x.(T)

// type assertion

v, ok = <-ch

// channel receive

As wit h var iable decl arat ions, we can assig n unwante d values to the blank identifier : _, err = io.Copy(dst, src) // discard byte count

_, ok = x.(T)

// check type but discard result

2.4.2. Assignability

As sig nment statements are an explicit for m of assig nment, but there are many places in a prog ram where an assig nment occ urs implicit ly: a function cal l implicitly assig ns the argument values to the cor responding parameter variables; a return st atement imp licitly assig ns the return op erands to the cor responding result var iables; and a lit eral expression for a composite type (§4.2) such as this slice:

medals := []string{"gold", "silver", "bronze"}

ptg16091132

implicitly assig ns each element, as if it had been writt en like this: medals[0] = "gold"

medals[1] = "silver"

medals[2] = "bronze"

The elements of maps and channel s, thoug h not ordinar y var iables, are als o su bjec t to simi lar implicit assig nments.

An assig nment, explicit or imp licit, is always legal if the lef t-hand side (the var iable) and the right-hand side (the value) have the same typ e. More general ly, the assig nment is legal only if the value is assig nab le to the typ e of the var iable.

The rule for assig nab ility has cas es for var ious typ es, so we’ll explain the relevant cas e as we introduce each new typ e. For the typ es we’ve dis cussed so far, the rules are simple: the typ es mu st exac tly match, and nil may be assig ned to any var iable of int erface or reference typ e.

Cons tants (§3.6) have more flexible rules for assig nabilit y that avoid the need for most explicit conv ersions.

Wh ether two values may be compare d with == and != is rel ate d to assig nabilit y: in any comparison, the firs t op erand must be assig nable to the typ e of the secon d op erand, or vice versa.

As wit h assig nabilit y, we’ll explain the relevant cas es for comp arab ility when we present each ne w type.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 2.5. TYPE DECLARATIONS

39

2.5. Type Declarations

The typ e of a var iable or expression defines the charac ter ist ics of the values it may take on, such as their size (number of bits or number of elements, perhaps), how the y are represent ed internal ly, the int rinsic operat ions that can be per for med on them, and the methods ass ociated wit h them.

In any program there are var iables that share the same represent ation but sig nify ver y dif ferent con cepts. For ins tance, an int could be used to represent a loop index, a timestamp, a file des crip tor, or a month; a float64 could represent a velocit y in meters per secon d or a temperature in one of several scales; and a string could represent a password or the name of a color.

A type de clarat ion defines a new name d ty pe that has the same un derly ing type as an exist ing type. The named typ e prov ides a way to sep arate dif ferent and perhaps incomp atible uses of the underly ing typ e so that the y can’t be mixed unintent ion ally.

type name underlying-type

Type declarat ions most often appear at package level, where the named typ e is visible throughout the package, and if the name is exp orted (it starts wit h an upp er-case letter), it’s accessible from other packages as wel l.

To illustrate typ e de clarat ions, let’s tur n the dif ferent temperature scales into dif ferent typ es: gopl.io/ch2/tempconv0

ptg16091132

// Package tempconv performs Celsius and Fahrenheit temperature computations.

package tempconv

import "fmt"

type Celsius float64

type Fahrenheit float64

const (

AbsoluteZeroC Celsius = -273.15

FreezingC

Celsius = 0

BoilingC

Celsius = 100

)

func CToF(c Celsius) Fahrenheit { return Fahrenheit(c*9/5 + 32) }

func FToC(f Fahrenheit) Celsius { return Celsius((f - 32) * 5 / 9) }

This package defines two typ es, Celsius and Fahrenheit, for the two units of temperature.

Ev en thoug h both have the same underly ing typ e, float64, the y are not the same typ e, so the y cannot be compare d or com bine d in arit hmetic expressions. Distinguishing the typ es makes it possible to avoid erro rs like inadver tently com bining temperatures in the two dif ferent scales; an explicit typ e conv ersion li ke Celsius(t) or Fahrenheit(t) is required to convert from a float64. Celsius(t) and Fahrenheit(t) are conversions, not function cal ls. They don’t ch ange the value or represent ation in any way, but the y make the change of meaning explicit.

On the other hand, the functions CToF and FToC conv ert bet ween the two scales; the y do return dif ferent values.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

40

CHAPTER 2. PROGRAM STRUCTURE

Fo r ever y type T, there is a cor responding conversion operat ion T(x) that conv erts the value x to typ e T. A conv ersion fro m on e type to another is allow ed if bot h have the same underly ing type, or if bot h are unnamed point er typ es that point to var iables of the same underly ing typ e; thes e conv ersions change the typ e but not the represent ation of the value. If x is assig nable to T, a conversion is per mitt ed but is usu ally redundant, Conv ersions are als o al lowe d between numer ic types, and bet ween str ing and som e slice typ es, as we will see in the next chapt er. These conversions may change the represent ation of the value. For ins tance, converting a floating-p oint number to an int eger dis cards any frac tional part, and converting a str ing to a []byte slice allo cates a copy of the str ing dat a. In any cas e, a conv ersion never fails at run time.

The underly ing typ e of a named typ e deter mines its str ucture and represent ation, and als o the set of int rinsic operat ions it sup por ts, which are the same as if the underly ing typ e had been us ed direc tly. That means that arithmetic operator s work the same for Celsius and Fahrenheit as the y do for float64, as you mig ht exp ect.

fmt.Printf("%g\n", BoilingC-FreezingC) // "100" °C

boilingF := CToF(BoilingC)

fmt.Printf("%g\n", boilingF-CToF(FreezingC)) // "180" °F

fmt.Printf("%g\n", boilingF-FreezingC)

// compile error: type mismatch

Comp arison operator s li ke == and < can als o be used to compare a value of a named typ e to ptg16091132

anot her of the same typ e, or to a value of the underly ing typ e. But two values of dif ferent name d types cannot be compare d direc tly :

var c Celsius

var f Fahrenheit

fmt.Println(c == 0)

// "true"

fmt.Println(f >= 0)

// "true"

fmt.Println(c == f)

// compile error: type mismatch

fmt.Println(c == Celsius(f)) // "true"!

No te the last cas e carefully. In spit e of its name, the typ e conv ersion Celsius(f) do es not ch ange the value of its argument, just its typ e. The test is true because c and f are bot h zero.

A named typ e may provide not ation al convenience if it helps avoid writing out complex typ es ov er and over again. The advantage is small when the underly ing typ e is simple like float64, but big for complic ated typ es, as we will see when we dis cuss str ucts.

Name d types als o make it possible to define new beh avior s for values of the typ e. These behavior s are express ed as a set of functions ass oci ated wit h the typ e, cal le d the typ e’s metho ds.

We’ll look at met hods in det ail in Chapt er 6 but will give a taste of the mech anism here.

The declarat ion below, in which the Celsius parameter c appears before the function name, asso ciates wit h the Celsius type a met hod named String that retur ns c’s numer ic value fo llowe d by °C:

func (c Celsius) String() string { return fmt.Sprintf("%g°C", c) }

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 2.6. PACKAGES AND FILES

41

Many typ es de clare a String method of this for m becaus e it control s how values of the typ e appear when print ed as a str ing by the fmt package, as we will see in Sec tion 7.1.

c := FToC(212.0)

fmt.Println(c.String()) // "100°C"

fmt.Printf("%v\n", c)

// "100°C"; no need to call String explicitly fmt.Printf("%s\n", c)

// "100°C"

fmt.Println(c)

// "100°C"

fmt.Printf("%g\n", c)

// "100"; does not call String

fmt.Println(float64(c)) // "100"; does not call String 2.6. Packages and Files

Packages in Go ser ve the same pur pos es as librar ies or modules in other langu ages, sup por ting mo dular ity, enc apsulat ion, sep arate compi lat ion, and reuse. The source code for a package resides in one or more .go files, usu ally in a direc tor y whos e name end s with the imp ort pat h; for ins tance, the files of the gopl.io/ch1/helloworld package are store d in direc tor y $GOPATH/src/gopl.io/ch1/helloworld.

Each package ser ves as a sep arate name space for its decl arat ions. Wit hin the image package, for example, the identifier Decode refers to a dif ferent function than does the same identifier ptg16091132

in the unicode/utf16 package. To refer to a function fro m outside its package, we must qu ali fy the identifier to make explicit whether we mean image.Decode or utf16.Decode.

Packages also let us hide infor mat ion by control ling which names are visible outside the package, or expor ted. In Go, a simple rule gov erns which identifiers are exp orted and which are not: expor ted identifiers start wit h an upp er-case letter.

To illustrate the basics, sup pos e that our temperature conversion sof tware has becom e popu lar and we want to make it avai lable to the Go community as a new package. How do we do that?

Let’s cre ate a package cal le d gopl.io/ch2/tempconv, a var iat ion on the pre vious example.

(Here we’ve made an exception to our usu al rule of numbering examples in sequence, so that the package pat h can be more realist ic.) The package its elf is store d in two files to show how de clarat ions in sep arate files of a package are accessed; in real life, a tiny package like this would need only one file.

We have put the declarat ions of the typ es, their cons tants, and their met hods in tempconv.go: gopl.io/ch2/tempconv

// Package tempconv performs Celsius and Fahrenheit conversions.

package tempconv

import "fmt"

type Celsius float64

type Fahrenheit float64

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

42

CHAPTER 2. PROGRAM STRUCTURE

const (

AbsoluteZeroC Celsius = -273.15

FreezingC

Celsius = 0

BoilingC

Celsius = 100

)

func (c Celsius) String() string

{ return fmt.Sprintf("%g°C", c) }

func (f Fahrenheit) String() string { return fmt.Sprintf("%g°F", f) }

and the conversion functions in conv.go:

package tempconv

// CToF converts a Celsius temperature to Fahrenheit.

func CToF(c Celsius) Fahrenheit { return Fahrenheit(c*9/5 + 32) }

// FToC converts a Fahrenheit temperature to Celsius.

func FToC(f Fahrenheit) Celsius { return Celsius((f - 32) * 5 / 9) }

Each file starts wit h a package de clarat ion that defines the package name. When the package is imp orted, its members are referred to as tempconv.CToF and so on. Package-le vel names li ke the typ es and con stants declare d in one file of a package are visible to all the other files of the package, as if the source code were all in a single file. Not e that tempconv.go imports fmt, but conv.go do es not, because it does not use anything fro m fmt.

Becaus e the package-le vel const names beg in with upp er-case letters, the y to o are accessible ptg16091132

with qualified names like tempconv.AbsoluteZeroC:

fmt.Printf("Brrrr! %v\n", tempconv.AbsoluteZeroC) // "Brrrr! -273.15°C"

To convert a Cel siu s temp erature to Fahren heit in a package that imp orts gopl.io/ch2/tempconv, we can write the fol low ing code:

fmt.Println(tempconv.CToF(tempconv.BoilingC)) // "212°F"

The doc comment (§10.7.4) immediate ly pre ceding the package de clarat ion documents the package as a whole. Convent ion ally, it should start wit h a summar y sent ence in the sty le illustrated. Only one file in each package should have a package doc comment. Extensive doc comments are often place d in a file of their own, convent ion ally cal le d doc.go.

Exercis e 2.1: Add typ es, cons tants, and functions to tempconv for pro cessing temperatures in the Kelvin scale, where zero Kelvin is −273.15°C and a dif ference of 1K has the same mag nitude as 1°C.

2.6.1. Imports

Wi thin a Go program, every package is identified by a unique str ing cal le d its import pat h.

Thes e are the str ings that app ear in an import de clarat ion like "gopl.io/ch2/tempconv".

The langu age specification doesn’t define where these str ings com e from or what the y me an; it’s up to the tools to int erpret them. When using the go to ol (Chapter 10), an import pat h denot es a direc tor y cont aining one or more Go source files that toget her make up the package.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 2.6. PACKAGES AND FILES

43

In addition to its import pat h, each package has a pack age nam e, which is the short (and not ne cessarily unique) name that app ears in its package de clarat ion. By convent ion, a package’s name match es the last seg ment of its import pat h, making it easy to pre dic t that the package name of gopl.io/ch2/tempconv is tempconv.

To use gopl.io/ch2/tempconv, we must imp ort it:

gopl.io/ch2/cf

// Cf converts its numeric argument to Celsius and Fahrenheit.

package main

import (

"fmt"

"os"

"strconv"

"gopl.io/ch2/tempconv"

)

func main() {

for _, arg := range os.Args[1:] {

t, err := strconv.ParseFloat(arg, 64)

if err != nil {

fmt.Fprintf(os.Stderr, "cf: %v\n", err)

os.Exit(1)

}

ptg16091132

f := tempconv.Fahrenheit(t)

c := tempconv.Celsius(t)

fmt.Printf("%s = %s, %s = %s\n",

f, tempconv.FToC(f), c, tempconv.CToF(c))

}

}

The imp ort declarat ion binds a short name to the imp orted package that may be used to refer to its cont ents throughout the file. The import ab ove lets us refer to names wit hin gopl.io/ch2/tempconv by using a qu alified ident ifier li ke tempconv.CToF. By defau lt, the short name is the package name—tempconv in this cas e—but an imp ort declarat ion may sp ecif y an alternat ive name to avoid a conflic t (§10.3).

The cf prog ram conv erts a single numer ic command-line argument to its value in bot h Celsiu s and Fahren heit:

$ go build gopl.io/ch2/cf

$ ./cf 32

32°F = 0°C, 32°C = 89.6°F

$ ./cf 212

212°F = 100°C, 212°C = 413.6°F

$ ./cf -40

-40°F = -40°C, -40°C = -40°F

It is an error to imp ort a package and then not refer to it. This che ck helps eliminate dep endencies that becom e unnecessary as the code evo l ves, although it can be a nuis ance dur ing

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

44

CHAPTER 2. PROGRAM STRUCTURE

debugging, since commenting out a line of code like log.Print("got here!") may remov e the sole reference to the package name log, causing the compi ler to emit an error. In this situation, you need to comment out or delete the unnecessary import.

Bett er still, use the golang.org/x/tools/cmd/goimports to ol, which aut omat ical ly ins erts and remov es packages fro m the imp ort declarat ion as necessary ; most editors can be configured to run goimports each time you save a file. Like the gofmt to ol, it als o pretty-prints Go source files in the canonic al format.

Exercis e 2.2: Wr ite a general-pur pos e unit-conv ersion program analogou s to cf that reads numb ers fro m its command-line arguments or fro m the stand ard inp ut if there are no arguments, and converts each number int o units like temperature in Cel siu s and Fahren heit, lengt h in feet and meters, weig ht in pound s and kilog rams, and the like.

2.6.2. Package Initialization

Package initializat ion beg ins by initializing package-le vel var iables in the order in which the y are declare d, except that dep endencies are res olved firs t: var a = b + c

// a initialized third, to 3

var b = f()

// b initialized second, to 2, by calling f

var c = 1

// c initialized first, to 1

ptg16091132

func f() int { return c + 1 }

If the package has multiple .go files, the y are initialize d in the order in which the files are given to the compi ler ; the go to ol sor ts .go files by name before inv oking the compi ler.

Each var iable decl are d at package level starts life wit h the value of its initializer expression, if any, but for som e var iables, like tables of dat a, an initializer expression may not be the simplest way to set its initial value. In that cas e, the init func tion mechanism may be simpler. Any file may cont ain any number of functions whose declarat ion is just func init() { /* ... */ }

Such init func tions can’t be cal le d or reference d, but other wis e they are nor mal functions.

Wi thin each file, init func tions are aut omat ical ly exe cut ed when the program starts, in the order in which the y are declare d.

One package is initialize d at a time, in the order of imp orts in the program, dependencies firs t, so a package p importing q can be sure that q is fully initialize d before p’s initializat ion beg ins.

Init ializat ion pro ceeds fro m the bot tom up; the main package is the last to be initialize d. In this manner, all packages are fully initialize d before the app lic ation’s main func tion beg ins.

The package below defines a function PopCount that retur ns the number of set bits, that is, bits whose value is 1, in a uint64 value, which is cal le d its popu lat ion count. It uses an init func tion to pre compute a table of results, pc, for each possible 8-bit value so that the PopCount func tion needn’t take 64 steps but can just retur n the sum of eig ht table lookups. (This is defi-nite ly not the fastest algor it hm for count ing bits, but it’s convenient for illustrat ing init

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 2.7. SCOPE

45

func tions, and for showing how to pre compute a table of values, which is often a useful prog ramming technique.)

gopl.io/ch2/popcount

package popcount

// pc[i] is the population count of i.

var pc [256]byte

func init() {

for i := range pc {

pc[i] = pc[i/2] + byte(i&1)

}

}

// PopCount returns the population count (number of set bits) of x.

func PopCount(x uint64) int {

return int(pc[byte(x>>(0*8))] +

pc[byte(x>>(1*8))] +

pc[byte(x>>(2*8))] +

pc[byte(x>>(3*8))] +

pc[byte(x>>(4*8))] +

pc[byte(x>>(5*8))] +

pc[byte(x>>(6*8))] +

pc[byte(x>>(7*8))])

ptg16091132

}

No te that the range lo op in init us es only the index; the value is unnecessary and thu s ne e d not be include d. The loop cou ld als o have been writt en as for i, _ := range pc {

We’ll see other uses of init func tions in the next sec tion and in Sec tion 10.5.

Exercis e 2.3: Re writ e PopCount to use a loop ins tead of a single expression. Compare the performance of the two versions. (Section 11.4 shows how to compare the per for mance of dif ferent imp lementation s systematically.)

Exercis e 2.4: Wr ite a version of PopCount that counts bits by shif ting its argument through 64

bit position s, testing the rig htmost bit each time. Compare its per for mance to the table-lo oku p version.

Exercis e 2.5: The expression x&(x-1) clears the rig htmost non-zero bit of x. Write a version of PopCount that counts bits by using this fac t, and ass ess its per for mance.

2.7. Scope

A declarat ion ass oci ates a name wit h a program entity, such as a function or a var iable. The scop e of a declarat ion is the par t of the source code where a use of the declare d name refers to that decl arat ion.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

46

CHAPTER 2. PROGRAM STRUCTURE

Don’t confuse scop e with lifet ime. The scope of a declarat ion is a reg ion of the program text; it is a compi le-t ime pro per ty. The lifet ime of a var iable is the range of time dur ing exe cut ion when the var iable can be refer red to by other par ts of the program; it is a run-t ime pro per ty.

A syntactic bl o ck is a sequence of statements enclos ed in braces like those that sur round the body of a function or loop. A name declare d inside a synt actic blo ck is not visible outside that block. The blo ck enclos es its declarat ions and deter mines their scope. We can generalize this notion of blo cks to include other gro upings of declarat ions that are not explicitly sur rounded by braces in the source code; we’ll cal l them all lexical blocks. There is a lexic al block for the entire source code, cal le d the univ ers e bl o ck; for each package; for each file; for each for, if, and switch st atement; for each cas e in a switch or select st atement; and, of cours e, for each explicit syntactic blo ck.

A declarat ion’s lexic al block deter mines its scope, which may be large or small. The declara-tion s of bui lt-in typ es, func tions, and con stants like int, len, and true are in the univers e block and can be refer red to throughout the ent ire program. Declarat ions outside any function, that is, at pack age level, can be refer red to fro m any file in the same package. Imp orted packages, such as fmt in the tempconv example, are declare d at the file level, so the y can be referred to fro m the same file, but not fro m anot her file in the same package wit hout another import. Many declarat ions, like that of the var iable c in the tempconv.CToF func tion, are local, so the y can be refer red to only fro m within the same function or perhaps just a par t of it.

ptg16091132

The scope of a control-flow lab el, as used by break, continue, and goto st atements, is the entire enclosing function.

A program may cont ain multiple declarat ions of the same name so lon g as each declarat ion is in a different lexic al block. For example, you can decl are a local variable wit h the same name as a package-le vel var iable. Or, as shown in Sec tion 2.3.3, you can decl are a function parameter cal le d new, even thoug h a function of this name is pre declare d in the univers e block. Don’t ov erdo it, thoug h; the larger the scope of the redeclarat ion, the more likely you are to sur prise the reader.

Wh en the compi ler encounters a reference to a name, it looks for a declarat ion, starting wit h the inner most enclosing lexic al block and wor king up to the univers e block. If the compi ler find s no declarat ion, it rep orts an ‘‘unde clare d name’’ er ror. If a name is declare d in bot h an outer blo ck and an inner blo ck, the inner decl arat ion will be found firs t. In that cas e, the inner decl arat ion is said to sh a dow or hi d e the out er one, mak ing it inaccessible: func f() {}

var g = "g"

func main() {

f := "f"

fmt.Println(f) // "f"; local var f shadows package-level func f fmt.Println(g) // "g"; package-level var

fmt.Println(h) // compile error: undefined: h

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 2.7. SCOPE

47

Wi thin a function, lexical blo cks may be neste d to arbit rar y depth, so one local decl arat ion can sh adow another. Most blo cks are cre ate d by control-flow con str ucts like if st atements and for lo ops. The program below has three dif ferent var iables cal le d x becaus e each declarat ion appears in a dif ferent lexic al block. (This example illustrates scope rules, not good sty le!) func main() {

x := "hello!"

for i := 0; i < len(x); i++ {

x := x[i]

if x != '!' {

x := x + 'A' - 'a'

fmt.Printf("%c", x) // "HELLO" (one letter per iteration)

}

}

}

The expressions x[i] and x + 'A' - 'a' each refer to a declarat ion of x from an out er blo ck; we’ll explain that in a mom ent. (No te that the latter expression is not equivalent to unicode.ToUpper.)

As mention ed above , not all lexic al blocks correspond to explicit brace-delimite d sequences of st atements; som e are merely imp lie d. The for lo op above cre ates two lexic al blocks: the explicit blo ck for the loop body, and an imp licit blo ck that addition ally enclos es the var iables ptg16091132

de clare d by the initializat ion claus e, such as i. The scope of a var iable decl are d in the imp licit block is the con dit ion, post-statement (i++), and body of the for st atement.

The example below als o has three var iables named x, each declare d in a different blo ck—on e in the function body, one in the for st atement’s blo ck, and one in the loop body—but only two of the blo cks are explicit:

func main() {

x := "hello"

for _, x := range x {

x := x + 'A' - 'a'

fmt.Printf("%c", x) // "HELLO" (one letter per iteration)

}

}

Like for lo ops, if st atements and switch st atements als o create imp licit blo cks in addition to their body blo cks. The code in the fol low ing if-else ch ain shows the scope of x and y: if x := f(); x == 0 {

fmt.Println(x)

} else if y := g(x); x == y {

fmt.Println(x, y)

} else {

fmt.Println(x, y)

}

fmt.Println(x, y) // compile error: x and y are not visible here

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

48

CHAPTER 2. PROGRAM STRUCTURE

The secon d if st atement is neste d within the firs t, so var iables decl are d within the firs t st atement’s initializer are visible wit hin the secon d. Simi lar rules apply to each cas e of a switch st atement: there is a blo ck for the con dit ion and a blo ck for each cas e body.

At the package level, the order in which declarat ions app ear has no effec t on their scope, so a de clarat ion may refer to its elf or to another that fol lows it, letting us declare recursive or mu tually rec ursive typ es and functions. The compi ler will rep ort an error if a con stant or var iable decl arat ion refers to its elf, how ever.

In this program:

if f, err := os.Open(fname); err != nil { // compile error: unused: f return err

}

f.ReadByte() // compile error: undefined f

f.Close()

// compile error: undefined f

the scope of f is just the if st atement, so f is not accessible to the statements that fol low, resu lting in compi ler erro rs. Dep ending on the compi ler, you may get an addition al error reporting that the local variable f was never used.

Thus it is often necessary to declare f before the con dit ion so that it is accessible after : f, err := os.Open(fname)

ptg16091132

if err != nil {

return err

}

f.ReadByte()

f.Close()

Yo u may be tempt ed to avoid decl aring f and err in the out er blo ck by mov ing the cal ls to ReadByte and Close inside an else block:

if f, err := os.Open(fname); err != nil {

return err

} else {

// f and err are visible here too

f.ReadByte()

f.Close()

}

but nor mal prac tice in Go is to deal wit h the error in the if block and then retur n, so that the successf ul exe cut ion pat h is not indente d.

Sh ort var iable decl arat ions demand an awareness of scope. Con sider the program below, which starts by obt aining its cur rent wor king direc tor y and sav ing it in a package-le vel var iable. This cou ld be don e by cal ling os.Getwd in function main, but it mig ht be better to sep arate this con cer n from the primary log ic, especi ally if fai ling to get the direc tor y is a fat al er ror.

The function log.Fatalf pr ints a message and cal ls os.Exit(1).

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 2.7. SCOPE

49

var cwd string

func init() {

cwd, err := os.Getwd() // compile error: unused: cwd

if err != nil {

log.Fatalf("os.Getwd failed: %v", err)

}

}

Since neither cwd nor err is already declare d in the init func tion’s blo ck, the := st atement de clares bot h of them as local variables. The inner decl arat ion of cwd makes the out er one inaccessible, so the statement does not update the package-le vel cwd var iable as intended.

Current Go compi lers detec t that the local cwd var iable is never used and rep ort this as an er ror, but the y are not str ictly required to per for m this che ck. Fur thermore, a minor change , such as the addition of a log ging statement that refers to the local cwd would defeat the che ck.

var cwd string

func init() {

cwd, err := os.Getwd() // NOTE: wrong!

if err != nil {

log.Fatalf("os.Getwd failed: %v", err)

}

log.Printf("Working directory = %s", cwd)

}

ptg16091132

The global cwd var iable remains uninitialize d, and the app arent ly nor mal log out put obfuscates the bug .

There are a number of ways to deal wit h this pot ent ial pro blem. The most direc t is to avoid :=

by declaring err in a sep arate var de clarat ion:

var cwd string

func init() {

var err error

cwd, err = os.Getwd()

if err != nil {

log.Fatalf("os.Getwd failed: %v", err)

}

}

We’ve now seen how packages, files, decl arat ions, and statements express the str ucture of prog rams. In the next two chapt ers, we’ll look at the str ucture of dat a.

This page intentionally left blank

ptg16091132

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

3

Basic Data Types

It’s all bits at the bot tom, of cours e, but computers operate fundament ally on fixe d-size numbers cal le d word s, which are int erprete d as integers, floating-p oint numbers, bit sets, or memor y addresses, then com bine d into larger aggregates that represent packets, pixel s, por tfo lios, ptg16091132

poetr y, and everything els e. Go offers a var iety of ways to organize dat a, with a spectrum of data typ es that at one end match the features of the hardware and at the other end provide what programmers need to conveniently represent complic ated dat a st ruc tures.

Go’s typ es fal l into four categor ies: basi c ty pes, ag gre gat e ty pes, reference types, and interface ty pes. Basic typ es, the topic of this chapt er, include numbers, str ings, and boole ans. Aggregate types—ar rays (§4.1) and str ucts (§4.4)—form more complic ated dat a types by com bining values of several simpler ones. Reference typ es are a diverse gro up that includes point ers (§2.3.2), slices (§4.2), maps (§4.3), functions (Chapter 5), and channel s (C hapter 8), but what the y have in common is that the y al l refer to program variables or state in direc tly, so that the effec t of an op erat ion app lie d to one reference is obs erve d by all copies of that reference. Final ly, we’ll tal k ab out int erface typ es in Chapt er 7.

3.1. Int egers

Go’s numer ic data typ es include several sizes of int egers, floating-p oint numbers, and complex numb ers. Each numer ic type deter mines the size and sig nedness of its values. Let’s beg in with integers.

Go provides bot h sig ned and unsig ned int eger arithmetic. There are four distinc t sizes of sig ned int egers—8, 16, 32, and 64 bits—repres ente d by the typ es int8, int16, int32, and int64, and cor responding unsig ned versions uint8, uint16, uint32, and uint64.

51

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

52

CHAPTER 3. BASIC DATA TYPES

There are als o two typ es called just int and uint that are the natural or most efficient size for sig ned and unsig ned int egers on a par tic ular platfor m; int is by far the most widely used numeric typ e. Bot h thes e types have the same size, eit her 32 or 64 bits, but one must not make assump tions about which; different compi lers may make dif ferent choices even on identical hard ware.

The typ e rune is an synonym for int32 and convent ion ally indic ates that a value is a Unico de co de point. The two names may be used int erc hange ably. Simi larly, the typ e byte is an synonym for uint8, and emp hasizes that the value is a pie ce of raw dat a rat her than a small numeric quant ity.

Final ly, there is an unsig ned int eger typ e uintptr, whose width is not specified but is sufficient to hold all the bits of a point er value. The uintptr type is used only for low-le vel prog ramming, such as at the bound ary of a Go program wit h a C librar y or an operat ing system. We’ll see examples of this when we deal wit h the unsafe package in Chapt er 13.

Regardless of their size, int, uint, and uintptr are dif ferent typ es from their explicitly size d siblings. Thus int is not the same typ e as int32, even if the natural size of int egers is 32 bits, and an explicit conversion is required to use an int value where an int32 is needed, and vice vers a.

Signed numbers are repres ente d in 2’s-comp lement for m, in which the hig h-order bit is reserved for the sig n of the number and the range of values of an n-bit number is fro m −2n−1

ptg16091132

to 2n−1−1. Unsig ned int egers use the full range of bits for non-negat ive values and thu s have the range 0 to 2n−1. For ins tance, the range of int8 is −128 to 127, where as the range of uint8 is 0 to 255.

Go’s binar y op erator s for arithmetic, logic, and comparison are liste d here in order of decre asing pre cedence:

*

/

%

<<

>>

&

&

+

-

|

^

==

!=

<

<=

>

>=

&&

||

There are only five level s of precedence for binary operator s. Operator s at the same level as-so ciate to the lef t, so parenthes es may be required for clarity, or to make the operator s evaluate in the int ended order in an expression like mask & (1 << 28).

Each operator in the firs t two lines of the table abov e, for ins tance +, has a corresponding assig nment operator li ke += that may be used to abbre viate an assig nment statement.

The int eger arithmetic operator s +, -, *, and / may be app lie d to int eger, floating-p oint, and comp lex numbers, but the remainder operator % applies only to int egers. The beh avior of % for negat ive numbers var ies across programming langu ages. In Go, the sig n of the remainder is always the same as the sig n of the div idend, so -5%3 and -5%-3 are bot h -2. The beh avior of /

dep ends on whether its operands are int egers, so 5.0/4.0 is 1.25, but 5/4 is 1 becaus e integer division trunc ates the result toward zero.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 3.1. INTEGERS

53

If the result of an arithmetic operat ion, whether signe d or unsig ned, has more bits than can be repres ente d in the result typ e, it is said to ov erflow. The hig h-order bits that do not fit are si lently dis carde d. If the original number is a sig ned typ e, the result cou ld be negat ive if the lef tmost bit is a 1, as in the int8 example here:

var u uint8 = 255

fmt.Println(u, u+1, u*u) // "255 0 1"

var i int8 = 127

fmt.Println(i, i+1, i*i) // "127 -128 1"

Tw o integers of the same typ e may be compare d using the binar y comp arison operator s below; the typ e of a comparison expression is a boole an.

==

equal to

!=

not equ al to

<

less than

<=

less than or equ al to

>

greater than

>=

greater than or equ al to

In fac t, al l values of basic typ e—boole ans, numbers, and str ings—are comp arab le, meaning that two values of the same typ e may be compare d using the == and != op erator s. Fur thermore , integers, floating-p oint numbers, and str ings are ordere d by the comparison operator s.

ptg16091132

The values of many other typ es are not comparable, and no other typ es are ordered. As we encounter each typ e, we’ll present the rules gov erning the comp arab ility of its values.

There are als o unar y addition and subtrac tion operator s:

+

unar y positive (no effec t)

-

unar y negat ion

Fo r integers, +x is a shorthand for 0+x and -x is a shorthand for 0-x; for floating-p oint and comp lex numbers, +x is just x and -x is the negat ion of x.

Go als o prov ides the fol low ing bit w ise binar y op erator s, the firs t four of which tre at their operands as bit patterns wit h no con cept of arithmetic car ry or sig n:

&

bit w ise AND

|

bit w ise OR

^

bit w ise XOR

&

bit cle ar (AND NOT)

<<

lef t shif t

>>

right shif t

The operator ^ is bit w ise exc lusive OR (XO R) when used as a binar y op erator, but when used as a unar y prefix operator it is bit w ise negat ion or complement; that is, it retur ns a value wit h each bit in its operand inv erted. The &^ op erator is bit cle ar (AND NOT): in the expression z = x &^ y, each bit of z is 0 if the cor responding bit of y is 1; other wis e it equ als the corresp onding bit of x.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

54

CHAPTER 3. BASIC DATA TYPES

The code below shows how bit w ise operat ions can be used to int erpret a uint8 value as a comp act and efficient set of 8 indep endent bits. It uses Printf’s %b verb to print a numb er’s binary dig its; 08 mo difies %b (an adverb!) to pad the result wit h zeros to exac tly 8 dig its.

var x uint8 = 1<<1 | 1<<5

var y uint8 = 1<<1 | 1<<2

fmt.Printf("%08b\n", x)

// "00100010", the set {1, 5}

fmt.Printf("%08b\n", y)

// "00000110", the set {1, 2}

fmt.Printf("%08b\n", x&y)

// "00000010", the intersection {1}

fmt.Printf("%08b\n", x|y)

// "00100110", the union {1, 2, 5}

fmt.Printf("%08b\n", x^y)

// "00100100", the symmetric difference {2, 5}

fmt.Printf("%08b\n", x&^y) // "00100000", the difference {5}

for i := uint(0); i < 8; i++ {

if x&(1<<i) != 0 { // membership test

fmt.Println(i) // "1", "5"

}

}

fmt.Printf("%08b\n", x<<1) // "01000100", the set {2, 6}

fmt.Printf("%08b\n", x>>1) // "00010001", the set {0, 4}

(S ection 6.5 shows an implementation of int eger sets that can be much big ger than a byte.) In the shif t op erat ions x<<n and x>>n, the n op erand deter mines the number of bit position s ptg16091132

to shif t and must be unsig ned; the x op erand may be unsig ned or sig ned. Arithmetically, a lef t shif t x<<n is equivalent to multiplic ation by 2n and a rig ht shif t x>>n is equivalent to the floor of div ision by 2n.

Left shif ts fill the vac ated bits wit h zeros, as do rig ht shif ts of unsig ned numbers, but rig ht shif ts of sig ned numbers fill the vac ated bits wit h copies of the sig n bit. For this reason, it is important to use unsig ned arithmetic when you’re tre ating an int eger as a bit pattern.

Although Go provides unsig ned numbers and arithmetic, we tend to use the sig ned int form even for quant ities that can’t be negat ive , such as the lengt h of an array, thoug h uint mig ht seem a more obv iou s ch oice. Indeed, the bui lt-in len func tion retur ns a sig ned int, as in this lo op which announces prize medals in reverse order :

medals := []string{"gold", "silver", "bronze"}

for i := len(medals) - 1; i >= 0; i-- {

fmt.Println(medals[i]) // "bronze", "silver", "gold"

}

The alt ernat ive would be cal amitous. If len returned an unsig ned number, then i to o would be a uint, and the con dit ion i >= 0 would always be true by definition. After the third iteration, in which i == 0, the i-- st atement wou ld cause i to becom e not −1, but the maximum uint value (for example, 264−1), and the evaluation of medals[i] would fai l at run time, or pani c (§5.9), by att emp ting to access an element outside the bound s of the slice.

Fo r this reason, unsig ned numbers tend to be used only when their bit w ise operator s or peculi ar ar ithmetic operator s are required, as when imp lementing bit sets, parsing binar y file

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 3.1. INTEGERS

55

formats, or for hashing and crypt ography. The y are typic ally not used for merely non-negat ive qu ant ities.

In general, an explicit conversion is required to convert a value fro m on e type to another, and binary operator s for arithmetic and log ic (except shif ts) mu st have operands of the same typ e.

Although this occ asionally results in longer expressions, it als o eliminates a whole class of problems and makes programs easier to underst and.

As an example fami liar fro m ot her cont exts, consider this sequence: var apples int32 = 1

var oranges int16 = 2

var compote int = apples + oranges // compile error

At tempt ing to compi le these three declarat ions pro duces an erro r mess age: invalid operation: apples + oranges (mismatched types int32 and int16) This typ e mismatch can be fixe d in several ways, most direc tly by converting everything to a common typ e:

var compote = int(apples) + int(oranges)

As des crib ed in Sec tion 2.5, for every typ e T, the conversion operat ion T(x) conv erts the value x to typ e T if the conversion is allow ed. Many int eger-to-integer conversions do not ent ail any ptg16091132

ch ange in value; the y ju st tel l the compi ler how to int erpret a value. But a conversion that narrows a big integer int o a smaller one, or a conversion fro m integer to floating-p oint or vice vers a, may change the value or los e precision:

f := 3.141 // a float64

i := int(f)

fmt.Println(f, i)

// "3.141 3"

f = 1.99

fmt.Println(int(f)) // "1"

Flo at to int eger conversion dis cards any frac tional par t, tr unc ating toward zero. You should avoid conversions in which the operand is out of range for the target typ e, because the beh avior dep ends on the imp lementation:

f := 1e100 // a float64

i := int(f) // result is implementation-dependent

Integer lit erals of any size and typ e can be writt en as ordinar y de cimal numbers, or as octal numb ers if the y begin wit h 0, as in 0666, or as hexade cimal if the y begin wit h 0x or 0X, as in 0xdeadbeef. Hex digits may be upp er or low er cas e. Nowad ays octal numbers seem to be us ed for exac tly one pur pos e—file per missions on POSIX systems—but hexade cimal numbers are widely used to emp hasize the bit pattern of a number over its numer ic value.

Wh en print ing numbers using the fmt package, we can cont rol the radix and for mat wit h the

%d, %o, and %x verbs, as shown in this example:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

56

CHAPTER 3. BASIC DATA TYPES

o := 0666

fmt.Printf("%d %[1]o %#[1]o\n", o) // "438 666 0666"

x := int64(0xdeadbeef)

fmt.Printf("%d %[1]x %#[1]x %#[1]X\n", x)

// Output:

// 3735928559 deadbeef 0xdeadbeef 0XDEADBEEF

No te the use of two fmt tr icks. Usu ally a Printf format str ing containing multiple % verbs would require the same number of ext ra op erands, but the [1] ‘‘adverbs’’ af ter % te ll Printf to us e the firs t op erand over and over again. Secon d, the # adverb for %o or %x or %X te lls Printf to emit a 0 or 0x or 0X prefix respectively.

Rune lit erals are writt en as a charac ter wit hin single quotes. The simplest example is an ASCII ch arac ter like 'a', but it’s possible to write any Unico de co de point eit her direc tly or wit h numeric escap es, as we will see shortly.

Runes are print ed wit h %c, or wit h %q if quoting is desired: ascii := 'a'

unicode := 'D'

newline := '\n'

fmt.Printf("%d %[1]c %[1]q\n", ascii)

// "97 a 'a'"

fmt.Printf("%d %[1]c %[1]q\n", unicode) // "22269 D 'D'"

fmt.Printf("%d %[1]q\n", newline)

// "10 '\n'"

ptg16091132

3.2. Floating-Point Numbers

Go provides two sizes of floating-p oint numbers, float32 and float64. Their arit hmetic prop erties are gov erned by the IEEE 754 stand ard imp lemente d by all moder n CPUs.

Values of these numer ic types range fro m tiny to huge. The limits of floating-p oint values can be found in the math package. The con stant math.MaxFloat32, the largest float32, is about 3.4e38, and math.MaxFloat64 is about 1.8e308. The smallest positive values are near 1.4e-45 and 4.9e-324, respectively.

A float32 prov ides approximately six decimal digits of pre cision, where as a float64

prov ides about 15 dig its; float64 should be preferred for most pur pos es becaus e float32

comp utation s acc umulate error rapid ly unless one is quite caref ul, and the smallest positive integer that cannot be exac tly represent ed as a float32 is not large: var f float32 = 16777216 // 1 << 24

fmt.Println(f == f+1)

// "true"!

Flo at ing-p oint numbers can be writt en lit eral ly using decimals, like this: const e = 2.71828 // (approximately)

Dig its may be omitt ed before the decimal point (.707) or after it (1.). Ver y smal l or ver y large numbers are better writt en in scientific not ation, wit h the letter e or E preceding the decimal expon ent:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 3.2. FLOATING-POINT NUMBERS

57

const Avogadro = 6.02214129e23

const Planck

= 6.62606957e-34

Flo at ing-p oint values are conveniently print ed wit h Printf’s %g verb, which cho oses the most comp act represent ation that has adequ ate pre cision, but for tables of dat a, the %e (exp onent) or %f (no exp onent) for ms may be more appro priate. All three verbs allow field width and numeric pre cision to be control led.

for x := 0; x < 8; x++ {

fmt.Printf("x = %d eA = %8.3f\n", x, math.Exp(float64(x)))

}

The code above prints the pow ers of e with thre e de cimal digits of pre cision, aligne d in an eig ht-charac ter field:

x = 0 eA =

1.000

x = 1 eA =

2.718

x = 2 eA =

7.389

x = 3 eA =

20.086

x = 4 eA =

54.598

x = 5 eA =

148.413

x = 6 eA =

403.429

x = 7 eA = 1096.633

In addition to a large col lec tion of the usu al mat hemat ical functions, the math package has ptg16091132

func tions for cre ating and detec ting the speci al values define d by IEEE 754: the positive and negat ive infinities, which represent numb ers of excessive mag nitude and the result of div ision by zero; and NaN (‘‘not a number’’), the result of such mat hemat ical ly dubious operat ions as 0/0 or Sqrt(-1).

var z float64

fmt.Println(z, -z, 1/z, -1/z, z/z) //

"0 -0 +Inf -Inf NaN"

The function math.IsNaN tests whether its argument is a not-a-number value, and math.NaN

returns such a value. It’s tempt ing to use NaN as a sentinel value in a numer ic comp utation, but testing whether a specific computation al result is equ al to NaN is fraug ht wit h peril becaus e any comparison wit h NaN always yields false:

nan := math.NaN()

fmt.Println(nan == nan, nan < nan, nan > nan) // "false false false"

If a function that retur ns a floating-p oint result mig ht fai l, it’s better to rep ort the fai lure sep arately, like this:

func compute() (value float64, ok bool) {

// ...

if failed {

return 0, false

}

return result, true

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

58

CHAPTER 3. BASIC DATA TYPES

The next program illustrates floating-p oint graphics comp utation. It plots a function of two var iables z = f(x, y) as a wire mesh 3-D sur face, using Scalable Vec tor Graphics (SVG), a st and ard XML not ation for line draw ings. Figure 3.1 shows an example of its out put for the func tion sin(r)/r, where r is sqrt(x*x+y*y).

ptg16091132

Figure 3.1. A sur face plot of the function sin(r)/r.

gopl.io/ch3/surface

// Surface computes an SVG rendering of a 3-D surface function.

package main

import (

"fmt"

"math"

)

const (

width, height = 600, 320

// canvas size in pixels

cells

= 100

// number of grid cells

xyrange

= 30.0

// axis ranges (-xyrange..+xyrange)

xyscale

= width / 2 / xyrange // pixels per x or y unit

zscale

= height * 0.4

// pixels per z unit

angle

= math.Pi / 6

// angle of x, y axes (=30°)

)

var sin30, cos30 = math.Sin(angle), math.Cos(angle) // sin(30°), cos(30°)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 3.2. FLOATING-POINT NUMBERS

59

func main() {

fmt.Printf("<svg xmlns='http://www.w3.org/2000/svg' "+

"style='stroke: grey; fill: white; stroke-width: 0.7' "+

"width='%d' height='%d'>", width, height)

for i := 0; i < cells; i++ {

for j := 0; j < cells; j++ {

ax, ay := corner(i+1, j)

bx, by := corner(i, j)

cx, cy := corner(i, j+1)

dx, dy := corner(i+1, j+1)

fmt.Printf("<polygon points='%g,%g %g,%g %g,%g %g,%g'/>\n", ax, ay, bx, by, cx, cy, dx, dy)

}

}

fmt.Println("</svg>")

}

func corner(i, j int) (float64, float64) {

// Find point (x,y) at corner of cell (i,j).

x := xyrange * (float64(i)/cells - 0.5)

y := xyrange * (float64(j)/cells - 0.5)

// Compute surface height z.

z := f(x, y)

ptg16091132

// Project (x,y,z) isometrically onto 2-D SVG canvas (sx,sy).

sx := width/2 + (x-y)*cos30*xyscale

sy := height/2 + (x+y)*sin30*xyscale - z*zscale

return sx, sy

}

func f(x, y float64) float64 {

r := math.Hypot(x, y) // distance from (0,0)

return math.Sin(r) / r

}

No tice that the function corner returns two values, the coordinates of the cor ner of the cel l.

The explanation of how the program wor ks requires only basic geometr y, but it’s fine to skip ov er it, since the point is to illustrate floating-p oint computation. The ess ence of the program is mapping bet ween three dif ferent coordinate systems, shown in Figure 3.2. The firs t is a 2-D

gr id of 100&100 cells identified by int eger coordinates (i, j), starting at (0, 0) in the far back corner. We plot fro m the back to the fro nt so that backg round polygons may be obs cured by foreground ones.

The secon d co ordinate system is a mesh of 3-D floating-p oint coordinates (x, y, z), where x and y are linear functions of i and j, transl ate d so that the origin is in the center, and scale d by the con stant xyrange. The heig ht z is the value of the sur face func tion f (x, y).

The third coordinate system is the 2-D image canvas, wit h (0, 0) in the top lef t corner. Points in this plane are denot ed (sx, sy). We use an isometr ic pro jec tion to map each 3-D point

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

60

CHAPTER 3. BASIC DATA TYPES

Figure 3.2. Three dif ferent coordinate systems.

(x, y, z) ont o the 2-D canvas. A point app ears far ther to the rig ht on the canvas the gre ater its x value or the sm all er its y value. And a point app ears far ther dow n the canvas the gre ater its x value or y value, and the smaller its z value. The ver tic al and hor izont al scale factors for x and y are der ive d from the sine and cosine of a 30° angle. The scale factor for z, 0.4, is an arbit rar y parameter.

Fo r each cel l in the 2-D grid, the main function computes the coordinates on the image canvas ptg16091132

of the four cor ners of the polygon AB CD, where B correspond s to (i, j) and A, C, and D are its neig hbors, then prints an SVG ins truction to draw it.

Exercis e 3.1: If the function f returns a non-finite float64 value, the SVG file will contain invalid <polygon> elements (although many SVG renderers handle this gracef ully). Modif y the program to skip invalid polygons.

Exercis e 3.2: Experiment wit h visu alizat ions of other functions fro m the math package. Can you pro duce an egg box, mogu ls, or a sadd le?

Exercis e 3.3: Color each polygon bas ed on its heig ht, so that the peaks are colored red (#ff0000) and the val leys blue (#0000ff).

Exercis e 3.4: Fo llowing the appro ach of the Lissajous example in Sec tion 1.7, cons truct a web server that computes sur faces and writes SVG dat a to the client. The ser ver must set the Content-Type he ader like this:

w.Header().Set("Content-Type", "image/svg+xml") (This step was not required in the Lissajous example because the ser ver uses stand ard heur ist ics to recog nize common for mats like PNG fro m the firs t 512 bytes of the respons e and generates the pro per header.) Allow the client to specif y values like heig ht, width, and color as HT TP re quest parameters.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 3.3. COMPLEX NUMBERS

61

3.3. Complex Numbers

Go provides two sizes of complex numbers, complex64 and complex128, whose components are float32 and float64 resp ectively. The bui lt-in function complex creates a complex number fro m its real and imaginary components, and the bui lt-in real and imag func tions ext ract thos e comp onents:

var x complex128 = complex(1, 2) // 1+2i

var y complex128 = complex(3, 4) // 3+4i

fmt.Println(x*y)

// "(-5+10i)"

fmt.Println(real(x*y))

// "-5"

fmt.Println(imag(x*y))

// "10"

If a floating-p oint lit eral or decimal integer lit eral is immediate ly fol low ed by i, such as 3.141592i or 2i, it becom es an im aginary lit era l, denot ing a complex number wit h a zero real comp onent:

fmt.Println(1i * 1i) // "(-1+0i)", i$ = -1

Un der the rules for con stant arithmetic, comp lex cons tants can be adde d to other cons tants (integer or floating point, real or imaginary), allowing us to write complex numbers natural ly, li ke 1+2i, or equivalently, 2i+1. The declarat ions of x and y ab ove can be simplified: ptg16091132

x := 1 + 2i

y := 3 + 4i

Comp lex numbers may be compare d for equ ality with == and !=. Two complex numbers are equal if their real par ts are equ al and their imag inar y parts are equ al.

The math/cmplx package provides librar y func tions for wor king wit h comp lex numbers, such as the complex squ are root and exp onent iat ion functions.

fmt.Println(cmplx.Sqrt(-1)) // "(0+1i)"

The fol low ing program uses complex128 ar ithmetic to generate a Mandelbro t set.

gopl.io/ch3/mandelbrot

// Mandelbrot emits a PNG image of the Mandelbrot fractal.

package main

import (

"image"

"image/color"

"image/png"

"math/cmplx"

"os"

)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

62

CHAPTER 3. BASIC DATA TYPES

func main() {

const (

xmin, ymin, xmax, ymax = -2, -2, +2, +2

width, height

= 1024, 1024

)

img := image.NewRGBA(image.Rect(0, 0, width, height))

for py := 0; py < height; py++ {

y := float64(py)/height*(ymax-ymin) + ymin

for px := 0; px < width; px++ {

x := float64(px)/width*(xmax-xmin) + xmin

z := complex(x, y)

// Image point (px, py) represents complex value z.

img.Set(px, py, mandelbrot(z))

}

}

png.Encode(os.Stdout, img) // NOTE: ignoring errors

}

func mandelbrot(z complex128) color.Color {

const iterations = 200

const contrast = 15

var v complex128

for n := uint8(0); n < iterations; n++ {

ptg16091132

v = v*v + z

if cmplx.Abs(v) > 2 {

return color.Gray{255 - contrast*n}

}

}

return color.Black

}

The two neste d lo ops iterate over each point in a 1024&1024 grays cale raster image represent-ing the −2 to +2 por tion of the complex plane. The program tests whether rep eatedly squ aring and adding the number that point represents event ual ly ‘‘es cap es’’ the circle of radius 2. If so, the point is shade d by the number of iterat ions it took to escap e. If not, the value belon gs to the Mandelbro t set, and the point remains black. Final ly, the program writes to its stand ard output the PNG-enco ded image of the iconic frac tal, shown in Figure 3.3.

Exercis e 3.5: Implement a full-color Mandelbro t set using the function image.NewRGBA and the typ e color.RGBA or color.YCbCr.

Exercis e 3.6: Su persamp ling is a technique to reduce the effec t of pixe lat ion by computing the color value at several points wit hin each pixel and tak ing the average. The simplest met hod is to div ide each pixel int o four ‘‘su bpixe ls.’’ Implement it.

Exercis e 3.7: Anot her simple frac tal uses Newton’s met hod to find complex solut ion s to a func tion such as z4−1 = 0. Shade each starting point by the number of iterat ions required to get clos e to one of the four roots. Color each point by the root it appro ach es.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 3.4. BOOLEANS

63

Figure 3.3. The Mandelbro t set.

Exercis e 3.8: Render ing frac tals at hig h zo om level s demands gre at arithmetic pre cision.

Implement the same frac tal using four dif ferent represent ation s of numbers: complex64, complex128, big.Float, and big.Rat. (The latter two typ es are found in the math/big package.

Float us es arbit rar y but bounded-pre cision floating-p oint; Rat us es unb ounde d-pre cision rat ional numbers.) How do the y comp are in per for mance and memor y us age? At what zoom ptg16091132

le vel s do render ing artifac ts become visible?

Exercis e 3.9: Wr ite a web ser ver that renders frac tals and writes the image dat a to the client.

Al low the client to specif y the x, y, and zoom values as parameters to the HTTP request.

3.4. Booleans

A value of typ e bool, or boolean, has only two possible values, true and false. The con dition s in if and for st atements are boole ans, and comparison operator s li ke == and < produce a boole an resu lt. The unary operator ! is log ical negat ion, so !true is false, or, one mig ht say, (!true==false)==true, alt hough as a matter of sty le, we always simplif y re dundant boole an expressions like x==true to x.

Boolean values can be com bine d with the && (AND) and || (OR) operator s, which have sh ort-circuit behavior : if the answer is already deter mined by the value of the lef t op erand, the rig ht op erand is not evaluate d, making it safe to write expressions like this: s != "" && s[0] == 'x'

where s[0] would panic if applie d to an emp ty str ing .

Since && has hig her pre cedence than || (mnemonic: && is boole an mu ltiplic ation, || is boole an addition), no parenthes es are required for con dit ion s of this for m:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

64

CHAPTER 3. BASIC DATA TYPES

if 'a' <= c && c <= 'z' ||

'A' <= c && c <= 'Z' ||

'0' <= c && c <= '9' {

// ...ASCII letter or digit...

}

There is no imp licit conversion fro m a boole an value to a numer ic value like 0 or 1, or vice vers a. It’s necessary to use an explicit if, as in

i := 0

if b {

i = 1

}

It mig ht be wor th writing a conversion function if this operat ion were needed often:

// btoi returns 1 if b is true and 0 if false.

func btoi(b bool) int {

if b {

return 1

}

return 0

}

The inv ers e op erat ion is so simple that it doesn’t war rant a function, but for symmetr y here it is:

ptg16091132

// itob reports whether i is non-zero.

func itob(i int) bool { return i != 0 }

3.5. Strings

A str ing is an immut able sequence of bytes. Str ings may cont ain arbit rar y data, including bytes wit h value 0, but usu ally the y cont ain human-readable text. Text str ings are convent ional ly int erprete d as UTF-8-enco ded sequences of Unico de co de points (runes), which we’ll explore in det ail ver y so on.

The bui lt-in len func tion retur ns the number of bytes (not runes) in a str ing , and the in dex op erat ion s[i] retr ieves the i-t h byte of str ing s, where 0 ≤ i < len(s).

s := "hello, world"

fmt.Println(len(s))

// "12"

fmt.Println(s[0], s[7]) // "104 119"

('h' and 'w')

At tempt ing to access a byte outside this range results in a panic: c := s[len(s)] // panic: index out of range

The i-t h byte of a str ing is not necessarily the i-t h ch ara cter of a str ing , becaus e the UTF-8

enco ding of a non-ASCII code point requires two or more bytes. Wor king wit h ch arac ters is discussed shortly.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 3.5. STRINGS

65

The substr ing op erat ion s[i:j] yields a new str ing con sisting of the bytes of the original str ing st arting at index i and continuing up to, but not including, the byte at index j. The result cont ains j-i bytes.

fmt.Println(s[0:5]) // "hello"

Again, a panic results if either index is out of bound s or if j is less than i.

Either or bot h of the i and j op erands may be omitt ed, in which cas e the defau lt values of 0

(t he start of the str ing) and len(s) (its end) are assume d, respectively.

fmt.Println(s[:5]) // "hello"

fmt.Println(s[7:]) // "world"

fmt.Println(s[:]) // "hello, world"

The + op erator makes a new str ing by con catenat ing two str ings: fmt.Println("goodbye" + s[5:]) // "goodbye, world"

St rings may be compare d with comparison operator s li ke == and <; the comparison is don e byte by byte, so the result is the natural lexicographic order ing.

St ring values are immut able: the byte sequence cont ained in a str ing value can never be ch ange d, thoug h of cours e we can assig n a new value to a str ing variab le. To app end one st ring to another, for ins tance, we can write

s := "left foot"

ptg16091132

t := s

s += ", right foot"

This does not modif y the str ing that s or iginally held but causes s to hold the new str ing formed by the += st atement; meanw hile, t st i l l cont ains the old str ing .

fmt.Println(s) // "left foot, right foot"

fmt.Println(t) // "left foot"

Since str ings are immut able, con str uctions that try to modif y a str ing’s dat a in place are not al lowe d:

s[0] = 'L' // compile error: cannot assign to s[0]

Immu tabilit y me ans that it is safe for two copies of a str ing to share the same underly ing memory, mak ing it che ap to copy str ings of any lengt h. Simi larly, a str ing s and a subst ring li ke s[7:] may safely share the same dat a, so the subst ring operat ion is als o ch eap. No new memory is allocated in eit her cas e. Figure 3.4 illustrates the arrangement of a str ing and two of its subst rings sharing the same underly ing byte array.

3.5.1. String Literals

A str ing value can be writt en as a st ring litera l, a sequence of bytes enclos ed in dou ble quotes:

"Hello, BF"

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

66

CHAPTER 3. BASIC DATA TYPES

Figure 3.4. The str ing "hello, world" and two subst rings.

Becaus e Go source files are always enco ded in UTF-8 and Go text str ings are convent ion ally interprete d as UTF-8, we can include Unico de co de points in str ing lit erals.

Wi thin a double-quote d st ring lit eral, es cap e sequ ences that beg in with a backslash \ can be us ed to ins ert arbit rar y byte values into the str ing . One set of escap es hand les ASCII cont rol co des li ke newline, car r iage retur n, and tab:

\a

‘‘aler t’’ or bel l

\b

backsp ace

\f

form feed

ptg16091132

\n

ne wline

\r

carriage retur n

\t

tab

\v

vertic al tab

\'

single quote (on ly in the rune lit eral '\'')

\"

double quote (on ly wit hin "..." literals)

\\

backsl ash

Arbit rar y bytes can also be include d in literal str ings using hexade cimal or octal escapes. A hexadecimal escap e is writt en \xhh, wit h exac tly two hexade cimal digits h (in upp er or low er case). An oc tal escap e is writt en \ooo with exac tly three octal digits o (0 through 7) not exce e ding \377. Bot h denot e a single byte wit h the specified value. Later, we’ll see how to enco de Unico de co de points numer ical ly in str ing lit erals.

A raw str ing litera l is writt en `...`, using backquotes ins tead of dou ble quotes. Wit hin a raw st ring lit eral, no escap e sequences are pro cessed; the contents are taken literal ly, including backsl ashes and newlines, so a raw str ing lit eral may spread over several lines in the program source. The only pro cessing is that car r iage retur ns are deleted so that the value of the str ing is the same on all platfor ms, including those that convent ion ally put car r iage retur ns in text files.

Raw str ing lit erals are a convenient way to write regu lar expressions, which tend to have lots of backsl ashes. They are als o us eful for HTML templ ates, JSON lit erals, command usage messages, and the like, which often extend over multiple lines.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 3.5. STRINGS

67

const GoUsage = `Go is a tool for managing Go source code.

Usage:

go command [arguments]

...`

3.5.2. Unicode

Long ago, life was simple and there was, at least in a parochi al view, only one charac ter set to de al with: ASCII, the American Stand ard Code for Infor mat ion Int erc hange . ASCII, or more precis ely US-ASCII, uses 7 bits to represent 128 ‘‘ch arac ters’’: the upp er- and low er-case letters of Eng lish, digits, and a var iety of punctuation and device-cont rol charac ters. For much of the early days of computing , this was adequ ate, but it lef t a ver y large frac tion of the wor ld’s popu lat ion unable to use their own writ ing systems in computers. Wit h the growt h of the Internet, dat a in myr iad langu ages has becom e much more common. How can this rich var i-et y be dealt wit h at all and, if possible, efficiently?

The answer is Unico de (unicode.org), which col lec ts all of the charac ters in all of the wor ld’s wr iting systems, plu s accents and other diacr itical marks, cont rol codes like tab and car r iage return, and plenty of esoterica, and assig ns each one a stand ard number cal le d a Un ico d e co d e point or, in Go ter minolog y, a rune.

ptg16091132

Unico de version 8 defines code points for over 120,000 charac ters in wel l ov er 100 langu ages and scr ipts. How are these represent ed in computer programs and dat a? The natural dat a type to hold a single rune is int32, and that’s what Go uses; it has the synonym rune for precis ely this pur pos e.

We cou ld represent a sequence of runes as a sequence of int32 values. In this represent ation, which is cal le d UTF-32 or UCS-4, the encoding of each Unico de co de point has the same size, 32 bits. This is simple and unifor m, but it uses much more space than necessary since most comp uter-re adable text is in ASCII, which requires only 8 bits or 1 byte per charac ter. All the ch arac ters in widespread use still number fewer than 65,536, which wou ld fit in 16 bits. Can we do better?

3.5.3. UTF-8

UTF-8 is a var iable-lengt h enco ding of Unico de co de points as bytes. UTF-8 was invent ed by Ken Thompson and Rob Pike, two of the cre ators of Go, and is now a Unico de st and ard . It us es bet ween 1 and 4 bytes to represent each rune, but only 1 byte for ASCII charac ters, and on ly 2 or 3 bytes for most runes in common use. The hig h-order bits of the firs t byte of the enco ding for a rune indic ate how many bytes fol low. A hig h-order 0 indic ates 7-bit ASCII, where each rune takes only 1 byte, so it is identical to convent ion al ASCII. A hig h-order 110

indic ates that the rune takes 2 bytes; the secon d byte beg ins wit h 10. Larger runes have analogous encodings.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

68

CHAPTER 3. BASIC DATA TYPES

0xxxxxx

runes 0−127 (ASCII)

11xxxxx 10xxxxxx

128−2047 (values <128 unus ed)

110xxxx 10xxxxxx 10xxxxxx

2048−65535 (values <2048 unus ed)

1110xxx 10xxxxxx 10xxxxxx 10xxxxxx

65536−0x10f f f f

(other values unus ed)

A var iable-lengt h enco ding pre cludes direc t indexing to access the n-t h ch arac ter of a str ing , but UTF-8 has many desirable pro per ties to compens ate. The encoding is compact, compatible wit h ASCII, and self-sy nchro nizing: it’s possible to find the beg inning of a charac ter by back ing up no more than three bytes. It’s als o a prefix code, so it can be decoded fro m lef t to right wit hout any ambiguity or lookahead. No rune’s encoding is a subst ring of any other, or even of a sequence of others, so you can searc h for a rune by just searc hing for its bytes, without wor rying about the pre ceding context. The lexicog raphic byte order equ als the Unico de co de point order, so sor ting UTF-8 wor ks natural ly. There are no emb edde d NUL (zero) bytes, which is convenient for prog ramming langu ages that use NUL to ter minate str ings.

Go source files are always enco ded in UTF-8, and UTF-8 is the preferred encoding for text st rings manipu lated by Go programs. The unicode package provides functions for wor king with indiv idu al runes (such as distinguishing letters fro m numb ers, or converting an upp er-case letter to a low er-case one), and the unicode/utf8 package provides functions for encoding and decoding runes as bytes using UTF-8.

Many Unico de ch arac ters are hard to typ e on a key board or to distinguish visually fro m sim-ptg16091132

ilar-lo oking ones; som e are even inv isible. Unico de es cap es in Go str ing lit erals allow us to sp ecif y them by their numer ic co de point value. There are two for ms, \uhhhh for a 16-bit value and \Uhhhhhhhh for a 32-bit value, where each h is a hexade cimal digit; the need for the 32-bit for m ar ises ver y inf requently. Each denot es the UTF-8 enco ding of the specified code point. Thus, for example, the fol low ing str ing lit erals all represent the same six-byte str ing:

"BF"

"\xe4\xb8\x96\xe7\x95\x8c"

"\u4e16\u754c"

"\U00004e16\U0000754c"

The three escap e sequences abov e prov ide alternat ive not ation s for the firs t st ring, but the values the y denot e are identical.

Unico de es cap es may als o be used in rune lit erals. Thes e thre e literals are equivalent:

'B'

'\u4e16'

'\U00004e16'

A rune whose value is less than 256 may be writt en wit h a single hexade cimal escape, such as

'\x41' for 'A', but for hig her values, a \u or \U es cap e mu st be used. Con sequently,

'\xe4\xb8\x96' is not a legal rune lit eral, even thoug h thos e thre e bytes are a valid UTF-8

enco ding of a single code point.

Than ks to the nice pro per ties of UTF-8, many str ing operat ions don’t require decoding. We can test whether one str ing contains another as a prefix:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 3.5. STRINGS

69

func HasPrefix(s, prefix string) bool {

return len(s) >= len(prefix) && s[:len(prefix)] == prefix

}

or as a suffix:

func HasSuffix(s, suffix string) bool {

return len(s) >= len(suffix) && s[len(s)-len(suffix):] == suffix

}

or as a subst ring:

func Contains(s, substr string) bool {

for i := 0; i < len(s); i++ {

if HasPrefix(s[i:], substr) {

return true

}

}

return false

}

using the same log ic for UTF-8-enco ded text as for raw bytes. This is not true for other enco dings. (The functions above are draw n from the strings package, thoug h its imp lementation of Contains us es a hashing technique to searc h more efficiently.) On the other hand, if we really care about the indiv idu al Unico de ch arac ters, we have to use ptg16091132

ot her mechanisms. Con sider the str ing fro m our ver y firs t example, which includes two East Asian charac ters. Figure 3.5 illustrates its represent ation in memor y. The str ing contains 13

bytes, but int erprete d as UTF-8, it encodes only nine code points or runes: import "unicode/utf8"

s := "Hello, BF"

fmt.Println(len(s))

// "13"

fmt.Println(utf8.RuneCountInString(s)) // "9"

To pro cess those charac ters, we need a UTF-8 deco der. The unicode/utf8 package provides on e that we can use like this:

for i := 0; i < len(s); {

r, size := utf8.DecodeRuneInString(s[i:])

fmt.Printf("%d\t%c\n", i, r)

i += size

}

Each cal l to DecodeRuneInString returns r, the rune its elf, and size, the number of bytes occupied by the UTF-8 enco ding of r. The size is used to update the byte index i of the next rune in the str ing . But this is clumsy, and we need loops of this kind all the time. For tunately, Go’s range lo op, when app lie d to a str ing , perfor ms UTF-8 deco ding imp licitly. The out put of the loop below is als o shown in Figure 3.5; not ice how the index jumps by more than 1 for each non-ASCII rune.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

70

CHAPTER 3. BASIC DATA TYPES

Figure 3.5. A range lo op decodes a UTF-8-enco ded str ing .

for i, r := range "Hello, BF" {

fmt.Printf("%d\t%q\t%d\n", i, r, r)

ptg16091132

}

We cou ld use a simple range lo op to count the number of runes in a str ing , li ke this: n := 0

for _, _ = range s {

n++

}

As wit h the other for ms of range lo op, we can omit the var iables we don’t need: n := 0

for range s {

n++

}

Or we can just cal l utf8.RuneCountInString(s).

We mention ed earlier that it is mostly a matter of convent ion in Go that text str ings are int erpreted as UTF-8-enco ded sequences of Unico de co de points, but for cor rec t us e of range lo ops on str ings, it’s more than a convent ion, it’s a necessity. What happens if we range over a st ring containing arbit rar y binary dat a or, for that matter, UTF-8 dat a cont aining error s?

Each time a UTF-8 deco der, whether explicit in a cal l to utf8.DecodeRuneInString or implicit in a range lo op, con sumes an unexp ected inp ut byte, it generates a speci al Unico de re placement chara cter, '\uFFFD', which is usu ally print ed as a white question mark inside a bl ack hexagon al or diamond-li ke shape (. When a program encounters this rune value, it’s of ten a sign that som e upst ream par t of the system that generated the str ing dat a has been

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 3.5. STRINGS

71

care less in its tre atment of text enco dings.

UTF-8 is exception ally convenient as an int erc hange for mat but wit hin a program runes may be more convenient because the y are of unifor m size and are thu s easi ly indexed in arrays and slices.

A []rune conv ersion app lie d to a UTF-8-enco ded str ing retur ns the sequence of Unico de co de points that the str ing encodes:

// "program" in Japanese katakana

s := ">+=@?"

fmt.Printf("% x\n", s) // "e3 83 97 e3 83 ad e3 82 b0 e3 83 a9 e3 83 a0"

r := []rune(s)

fmt.Printf("%x\n", r)

// "[30d7 30ed 30b0 30e9 30e0]"

(The verb % x in the firs t Printf inserts a space bet ween each pair of hex digits.) If a slice of runes is converted to a str ing , it produces the con catenat ion of the UTF-8 enco dings of each rune:

fmt.Println(string(r)) // ">+=@?"

Conv erting an int eger value to a str ing int erprets the int eger as a rune value, and yields the UTF-8 represent ation of that rune:

fmt.Println(string(65))

// "A", not "65"

ptg16091132

fmt.Println(string(0x4eac)) // "C"

If the rune is invalid, the replacement charac ter is subst itute d: fmt.Println(string(1234567)) // "("

3.5.4. Strings and Byte Slices

Fo ur stand ard packages are par tic ularly imp ortant for manipu lat ing str ings: bytes, strings, strconv, and unicode. The strings package provides many functions for searc hing, replacing, comparing , tr imming, split ting, and joining str ings.

The bytes package has similar functions for manipu lat ing slices of bytes, of typ e []byte, which share som e prop erties wit h st rings. Because str ings are immut able, bui lding up str ings incrementally can invo l ve a lot of allocat ion and copying . In such cas es, it’s more efficient to us e the bytes.Buffer type, which we’ll show in a mom ent.

The strconv package provides functions for converting boole an, integer, and floating-p oint values to and fro m their str ing represent ation s, and functions for quoting and unquoting st rings.

The unicode package provides functions like IsDigit, IsLetter, IsUpper, and IsLower for cl assif ying runes. Each function takes a single rune argument and retur ns a boole an. Conv ersion functions like ToUpper and ToLower conv ert a rune int o the given cas e if it is a letter. All thes e func tions use the Unico de st and ard categor ies for letters, dig its, and so on. The strings

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

72

CHAPTER 3. BASIC DATA TYPES

package has similar functions, als o called ToUpper and ToLower, that retur n a new str ing wit h the specified transfor mat ion app lie d to each charac ter of the original str ing .

The basename func tion below was inspired by the Unix shel l ut i lit y of the same name. In our version, basename(s) removes any prefix of s that looks like a file system pat h with components sep arated by slashes, and it remov es any suffix that looks like a file typ e: fmt.Println(basename("a/b/c.go")) // "c"

fmt.Println(basename("c.d.go"))

// "c.d"

fmt.Println(basename("abc"))

// "abc"

The firs t version of basename do es al l the wor k without the help of librar ies: gopl.io/ch3/basename1

// basename removes directory components and a .suffix.

// e.g., a => a, a.go => a, a/b/c.go => c, a/b.c.go => b.c func basename(s string) string {

// Discard last '/' and everything before.

for i := len(s) - 1; i >= 0; i-- {

if s[i] == '/' {

s = s[i+1:]

break

}

}

// Preserve everything before last '.'.

ptg16091132

for i := len(s) - 1; i >= 0; i-- {

if s[i] == '.' {

s = s[:i]

break

}

}

return s

}

A simpler version uses the strings.LastIndex librar y func tion: gopl.io/ch3/basename2

func basename(s string) string {

slash := strings.LastIndex(s, "/") // -1 if "/" not found s = s[slash+1:]

if dot := strings.LastIndex(s, "."); dot >= 0 {

s = s[:dot]

}

return s

}

The path and path/filepath packages provide a more general set of functions for manip-ulat ing hierarchic al names. The path package wor ks wit h sl ash-delimite d paths on any platform. It shouldn’t be used for file names, but it is appro priate for other domains, like the pat h comp onent of a URL. By contrast, path/filepath manipu lates file names using the rules for the host platfor m, such as /foo/bar for POSIX or c:\foo\bar on Micros oft Windows.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 3.5. STRINGS

73

Let’s continue wit h anot her subst ring example. The task is to take a str ing represent ation of an integer, such as "12345", and ins ert commas every three places, as in "12,345". This version on ly wor ks for int egers; handling floating-p oint numbers is lef t as a exercise.

gopl.io/ch3/comma

// comma inserts commas in a non-negative decimal integer string.

func comma(s string) string {

n := len(s)

if n <= 3 {

return s

}

return comma(s[:n-3]) + "," + s[n-3:]

}

The argument to comma is a str ing . If its lengt h is less than or equ al to 3, no comma is necessary. Other wis e, comma calls its elf rec ursively wit h a subst ring con sisting of all but the last thre e ch arac ters, and app end s a comma and the last three charac ters to the result of the rec ursive cal l.

A str ing contains an array of bytes that, once cre ate d, is immut able. By contrast, the elements of a byte slice can be fre ely modified.

ptg16091132

St rings can be converted to byte slices and back again: s := "abc"

b := []byte(s)

s2 := string(b)

Conceptu ally, the []byte(s) conv ersion allocates a new byte array holding a copy of the bytes of s, and yields a slice that references the ent irety of that array. An opt imizing compi ler may be able to avoid the allocat ion and copying in som e cases, but in general copy ing is required to ensure that the bytes of s remain unch ange d even if those of b are subsequently modified. The conv ersion fro m byte slice back to str ing wit h string(b) also makes a copy, to ens ure immut abi lit y of the resulting str ing s2.

To avoid conv ersions and unnecessary memor y al location, many of the utilit y func tions in the bytes package direc tly paral lel their counterpar ts in the strings package. For example, here are half a dozen functions fro m strings:

func Contains(s, substr string) bool

func Count(s, sep string) int

func Fields(s string) []string

func HasPrefix(s, prefix string) bool

func Index(s, sep string) int

func Join(a []string, sep string) string

and the cor responding ones fro m bytes:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

74

CHAPTER 3. BASIC DATA TYPES

func Contains(b, subslice []byte) bool

func Count(s, sep []byte) int

func Fields(s []byte) [][]byte

func HasPrefix(s, prefix []byte) bool

func Index(s, sep []byte) int

func Join(s [][]byte, sep []byte) []byte

The only dif ference is that str ings have been replace d by byte slices.

The bytes package provides the Buffer type for efficient manipu lat ion of byte slices. A Buffer st arts out emp ty but grows as dat a of typ es li ke string, byte, and []byte are writt en to it. As the example below shows, a bytes.Buffer var iable requires no initializat ion because its zero value is usable:

gopl.io/ch3/printints

// intsToString is like fmt.Sprintf(values) but adds commas.

func intsToString(values []int) string {

var buf bytes.Buffer

buf.WriteByte('[')

for i, v := range values {

if i > 0 {

buf.WriteString(", ")

}

fmt.Fprintf(&buf, "%d", v)

ptg16091132

}

buf.WriteByte(']')

return buf.String()

}

func main() {

fmt.Println(intsToString([]int{1, 2, 3})) // "[1, 2, 3]"

}

Wh en app ending the UTF-8 enco ding of an arbit rar y rune to a bytes.Buffer, it’s best to use bytes.Buffer’s WriteRune method, but WriteByte is fine for ASCII charac ters such as '['

and ']'.

The bytes.Buffer type is ext remely versatile, and when we dis cuss int erfaces in Chapt er 7, we’ll see how it may be used as a replacement for a file whene ver an I/O function requires a sin k for bytes (io.Writer) as Fprintf do es ab ove , or a source of bytes (io.Reader).

Exercis e 3.10: Wr ite a non-rec ursive version of comma, using bytes.Buffer instead of str ing conc atenation.

Exercis e 3.11: En hance comma so that it deals cor rec tly wit h floating-p oint numbers and an opt ion al sig n.

Exercis e 3.12: Wr ite a function that rep orts whether two str ings are anagrams of each other, that is, the y cont ain the same letters in a dif ferent order.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 3.6. CONSTANTS

75

3.5.5. Conversions between Strings and Numbers

In addition to conversions bet ween str ings, runes, and bytes, it’s often necessary to convert between numer ic values and their str ing represent ation s. This is don e with func tions fro m the strconv package.

To convert an int eger to a str ing , on e opt ion is to use fmt.Sprintf; another is to use the function strconv.Itoa (‘‘integer to ASCII’’):

x := 123

y := fmt.Sprintf("%d", x)

fmt.Println(y, strconv.Itoa(x)) // "123 123"

FormatInt and FormatUint can be used to for mat numbers in a dif ferent bas e: fmt.Println(strconv.FormatInt(int64(x), 2)) // "1111011"

The fmt.Printf verbs %b, %d, %u, and %x are often more convenient than Format func tions, especi ally if we want to include addition al infor mat ion besides the number: s := fmt.Sprintf("x=%b", x) // "x=1111011"

To parse a str ing represent ing an int eger, use the strconv func tions Atoi or ParseInt, or ParseUint for unsig ned int egers:

x, err := strconv.Atoi("123")

// x is an int

ptg16091132

y, err := strconv.ParseInt("123", 10, 64) // base 10, up to 64 bits The third argument of ParseInt gives the size of the int eger typ e that the result must fit int o; for example, 16 imp lies int16, and the speci al value of 0 imp lies int. In any cas e, the typ e of the result y is always int64, which you can then convert to a smaller typ e.

Sometimes fmt.Scanf is usef ul for parsing inp ut that con sists of orderly mixtures of str ings and numbers all on a single line, but it can be inflexible, esp eci ally when handling incomplete or irregu lar input.

3.6. Constants

Cons tants are expressions whose value is known to the compi ler and whose evaluation is guarante e d to occ ur at comp ile time, not at run time. The underly ing typ e of every con stant is a basic typ e: boole an, st ring, or number.

A const de clarat ion defines named values that look syntactic ally like var iables but whose value is con stant, which pre vents accidental (or nefar ious) changes dur ing program exec ution.

Fo r inst ance, a con stant is more appro priate than a var iable for a mat hemat ical cons tant like pi, since its value won’t change:

const pi = 3.14159 // approximately; math.Pi is a better approximation As wit h var iables, a sequence of con stants can appear in one declarat ion; this wou ld be appropriate for a gro up of rel ate d values:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

76

CHAPTER 3. BASIC DATA TYPES

const (

e

= 2.71828182845904523536028747135266249775724709369995957496696763

pi = 3.14159265358979323846264338327950288419716939937510582097494459

)

Many computation s on con stants can be completely evaluate d at comp ile time, reducing the work necessary at run time and enabling other comp iler opt imizat ions. Error s ordinar ily detec ted at run time can be rep orted at compi le time when their operands are con stants, such as integer div ision by zero, str ing indexing out of bound s, and any floating-p oint operat ion that wou ld result in a non-finite value.

The results of all arithmetic, logic al, and comparison operat ions app lie d to con stant operands are thems elves cons tants, as are the results of conversions and cal ls to cer tain bui lt-in function s such as len, cap, real, imag, complex, and unsafe.Sizeof (§13.1).

Since their values are known to the compi ler, con stant expressions may appear in typ es, sp ecific ally as the lengt h of an array typ e:

const IPv4Len = 4

// parseIPv4 parses an IPv4 address (d.d.d.d).

func parseIPv4(s string) IP {

var p [IPv4Len]byte

// ...

ptg16091132

}

A con stant declarat ion may specif y a typ e as wel l as a value, but in the abs ence of an explicit type, the typ e is infer red fro m the expression on the rig ht-hand side. In the fol low ing , time.Duration is a name d type whose underly ing typ e is int64, and time.Minute is a const ant of that typ e. Bot h of the con stants declare d below thu s have the typ e time.Duration as we ll, as reveale d by %T:

const noDelay time.Duration = 0

const timeout = 5 * time.Minute

fmt.Printf("%T %[1]v\n", noDelay)

// "time.Duration 0"

fmt.Printf("%T %[1]v\n", timeout)

// "time.Duration 5m0s

fmt.Printf("%T %[1]v\n", time.Minute) // "time.Duration 1m0s"

Wh en a sequence of con stants is declare d as a gro up, the rig ht-hand side expression may be omit ted for all but the firs t of the gro up, imp l ying that the pre vious expression and its typ e should be used again. For example:

const (

a = 1

b

c = 2

d

)

fmt.Println(a, b, c, d) // "1 1 2 2"

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 3.6. CONSTANTS

77

This is not ver y us eful if the imp licitly copie d right-hand side expression always evaluates to the same thing. But what if it cou ld var y? This brings us to iota.

3.6.1. The Constant Generator iota

A const de clarat ion may use the constant generat o r iota, which is used to cre ate a sequence of rel ate d values wit hout spelling out each one explicitly. In a const de clarat ion, the value of iota begins at zero and increments by one for each item in the sequence.

Here’s an example fro m the time package, which defines named con stants of typ e Weekday for the days of the week, starting wit h zero for Sunday. Typ es of this kind are often cal le d enu-merati ons, or enums for short.

type Weekday int

const (

Sunday Weekday = iota

Monday

Tuesday

Wednesday

Thursday

Friday

Saturday

ptg16091132

)

This declares Sunday to be 0, Monday to be 1, and so on.

We can use iota in more complex expressions too, as in this example fro m the net package where each of the low est 5 bits of an unsig ned int eger is given a dist inc t name and boole an interpretation:

type Flags uint

const (

FlagUp Flags = 1 << iota // is up

FlagBroadcast

// supports broadcast access capability

FlagLoopback

// is a loopback interface

FlagPointToPoint

// belongs to a point-to-point link

FlagMulticast

// supports multicast access capability

)

As iota increments, each con stant is assig ned the value of 1 << iota, which evaluates to successive pow ers of two, each cor responding to a single bit. We can use these con stants wit hin func tions that test, set, or cle ar on e or more of these bits: gopl.io/ch3/netflag

func IsUp(v Flags) bool

{ return v&FlagUp == FlagUp }

func TurnDown(v *Flags)

{ *v &^= FlagUp }

func SetBroadcast(v *Flags) { *v |= FlagBroadcast }

func IsCast(v Flags) bool

{ return v&(FlagBroadcast|FlagMulticast) != 0 }

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

78

CHAPTER 3. BASIC DATA TYPES

func main() {

var v Flags = FlagMulticast | FlagUp

fmt.Printf("%b %t\n", v, IsUp(v)) // "10001 true"

TurnDown(&v)

fmt.Printf("%b %t\n", v, IsUp(v)) // "10000 false"

SetBroadcast(&v)

fmt.Printf("%b %t\n", v, IsUp(v))

// "10010 false"

fmt.Printf("%b %t\n", v, IsCast(v)) // "10010 true"

}

As a more complex example of iota, this declarat ion names the pow ers of 1024: const (

_ = 1 << (10 * iota)

KiB // 1024

MiB // 1048576

GiB // 1073741824

TiB // 1099511627776

(exceeds 1 << 32)

PiB // 1125899906842624

EiB // 1152921504606846976

ZiB // 1180591620717411303424

(exceeds 1 << 64)

YiB // 1208925819614629174706176

)

The iota me chanism has its limits. For example, it’s not possible to generate the more fami l-ptg16091132

iar pow ers of 1000 (KB, MB, and so on) because there is no exp onent iat ion operator.

Exercis e 3.13: Wr ite const de clarat ions for KB, MB, up through YB as compactly as you can.

3.6.2. Untyped Constants

Cons tants in Go are a bit unu sual. Alt hough a con stant can have any of the basic dat a types li ke int or float64, including named basic typ es li ke time.Duration, many con stants are not committ ed to a par tic ular typ e. The compi ler represents these uncommitt ed con stants with much gre ater numer ic precision than values of basic typ es, and arithmetic on them is more pre cis e than machine arithmetic; you may assume at least 256 bits of pre cision. There are six flavor s of these uncommitt ed con stants, cal le d unty ped boole an, untyped int eger, untyped rune, unt yped floating-p oint, untyped complex, and unt yped str ing .

By defer r ing this commitment, untyped con stants not only ret ain their higher pre cision unt i l later, but the y can par ticipate in many more expressions than committ ed con stants wit hout re quir ing conversions. For example, the values ZiB and YiB in the example above are too big to store in any int eger var iable, but the y are leg itimate con stants that may be used in expressions like this one:

fmt.Println(YiB/ZiB) // "1024"

As another example, the floating-p oint con stant math.Pi may be used where ver any floating-point or complex value is needed:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 3.6. CONSTANTS

79

var x float32 = math.Pi

var y float64 = math.Pi

var z complex128 = math.Pi

If math.Pi had been committ ed to a specific typ e such as float64, the result wou ld not be as precis e, and typ e conv ersions would be required to use it when a float32 or complex128

value is wante d:

const Pi64 float64 = math.Pi

var x float32 = float32(Pi64)

var y float64 = Pi64

var z complex128 = complex128(Pi64)

Fo r literals, synt ax deter mines flavor. The lit erals 0, 0.0, 0i, and '\u0000' al l denot e const ants of the same value but dif ferent flavor s: unt yped int eger, unt yped floating-p oint, untyped comp lex, and unt yped rune, respectively. Simi larly, true and false are unt yped boole ans and st ring lit erals are unt yped str ings.

Re call that / may represent int eger or floating-p oint div ision dep ending on its operands.

Cons equently, the choice of lit eral may affe ct the result of a con stant div ision expression: var f float64 = 212

ptg16091132

fmt.Println((f - 32) * 5 / 9)

// "100"; (f - 32) * 5 is a float64

fmt.Println(5 / 9 * (f - 32))

// "0";

5/9 is an untyped integer, 0

fmt.Println(5.0 / 9.0 * (f - 32)) // "100"; 5.0/9.0 is an untyped float On ly con stants can be unt yped. When an unt yped con stant is assig ned to a var iable, as in the firs t st atement below, or app ears on the rig ht-hand side of a var iable decl arat ion wit h an explicit typ e, as in the other three statements, the con stant is imp licitly converted to the typ e of that var iable if possible.

var f float64 = 3 + 0i // untyped complex -> float64

f = 2

// untyped integer -> float64

f = 1e123

// untyped floating-point -> float64

f = 'a'

// untyped rune -> float64

The statements above are thu s equivalent to these:

var f float64 = float64(3 + 0i)

f = float64(2)

f = float64(1e123)

f = float64('a')

Wh ether implicit or explicit, conv erting a con stant fro m on e type to another requires that the target typ e can represent the original value. Rounding is allow ed for real and complex floating-p oint numbers:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

80

CHAPTER 3. BASIC DATA TYPES

const (

deadbeef = 0xdeadbeef // untyped int with value 3735928559

a = uint32(deadbeef) // uint32 with value 3735928559

b = float32(deadbeef) // float32 with value 3735928576 (rounded up) c = float64(deadbeef) // float64 with value 3735928559 (exact) d = int32(deadbeef)

// compile error: constant overflows int32

e = float64(1e309)

// compile error: constant overflows float64

f = uint(-1)

// compile error: constant underflows uint

)

In a var iable decl arat ion wit hout an explicit typ e (including short var iable decl arat ions), the flavor of the unt yped con stant imp licitly deter mines the defau lt typ e of the var iable, as in these examples:

i := 0

// untyped integer;

implicit int(0)

r := '\000' // untyped rune;

implicit rune('\000')

f := 0.0

// untyped floating-point; implicit float64(0.0)

c := 0i

// untyped complex;

implicit complex128(0i)

No te the asy mmet ry: unt yped int egers are converted to int, whose size is not guarante e d, but untyped floating-p oint and complex numbers are converted to the explicitly size d types float64 and complex128. The langu age has no unsize d float and complex types analogou s to unsize d int, because it is ver y difficult to write cor rec t numerical algor it hms wit hout know ing the size of one’s floating-p oint dat a types.

ptg16091132

To give the var iable a different typ e, we must explicitly convert the unt yped con stant to the desired typ e or state the desired typ e in the var iable decl arat ion, as in these examples: var i = int8(0)

var i int8 = 0

Thes e defau lts are par tic ularly imp ortant when converting an unt yped con stant to an int erface value (see Chapter 7) since the y deter mine its dynamic typ e.

fmt.Printf("%T\n", 0)

// "int"

fmt.Printf("%T\n", 0.0)

// "float64"

fmt.Printf("%T\n", 0i)

// "complex128"

fmt.Printf("%T\n", '\000') // "int32" (rune) We’ve now cov ere d the basic dat a types of Go. The next step is to show how the y can be combine d into larger gro upings like arrays and str ucts, and then int o data str uctures for solv ing re al programming pro blems; that is the topic of Chapt er 4.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

4

Composite Types

In Chapt er 3 we dis cussed the basic typ es that ser ve as bui lding blo cks for dat a st ruc tures in a Go program; the y are the atoms of our univers e. In this chapt er, we’ll take a look at comp osite types, the mole cules create d by com bining the basic typ es in various ways. We’ll tal k ab out ptg16091132

four such typ es—ar rays, slices, maps, and str ucts—and at the end of the chapt er, we’ll show how str uctured dat a using these typ es can be encoded as and parsed fro m JSON dat a and used to generate HTML fro m temp lates.

Ar rays and str ucts are ag gre gat e types; their values are con catenat ions of other values in memor y. Arrays are hom ogene ous—their elements all have the same typ e—whereas str ucts are heterogene ous. Bot h ar rays and str ucts are fixe d size. In contrast, slices and maps are dy namic dat a st ruc tures that grow as values are adde d.

4.1. Arrays

An array is a fixe d-lengt h sequence of zero or more elements of a par tic ular typ e. Because of their fixe d lengt h, ar rays are rarely used direc tly in Go. Slices, which can grow and shrink, are much more versatile, but to underst and slices we must underst and arrays firs t.

In div idu al array elements are accessed wit h the convent ion al subscript not ation, where su bscripts run fro m zero to one less than the array lengt h. The bui lt-in function len returns the number of elements in the array.

var a [3]int

// array of 3 integers

fmt.Println(a[0])

// print the first element

fmt.Println(a[len(a)-1]) // print the last element, a[2]

81

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

82

CHAPTER 4. COMPOSITE TYPES

// Print the indices and elements.

for i, v := range a {

fmt.Printf("%d %d\n", i, v)

}

// Print the elements only.

for _, v := range a {

fmt.Printf("%d\n", v)

}

By defau lt, the elements of a new array variable are initial ly set to the zero value for the element typ e, which is 0 for numbers. We can use an ar ray lit era l to initialize an array wit h a list of values:

var q [3]int = [3]int{1, 2, 3}

var r [3]int = [3]int{1, 2}

fmt.Println(r[2]) // "0"

In an array literal, if an ellipsis ‘‘...’’ appears in place of the lengt h, the array lengt h is determined by the number of initializers. The definition of q can be simplified to q := [...]int{1, 2, 3}

fmt.Printf("%T\n", q) // "[3]int"

The size of an array is par t of its typ e, so [3]int and [4]int are dif ferent typ es. The size ptg16091132

mu st be a con stant expression, that is, an expression whose value can be compute d as the prog ram is being compi led.

q := [3]int{1, 2, 3}

q = [4]int{1, 2, 3, 4} // compile error: cannot assign [4]int to [3]int As we’ll see, the lit eral synt ax is simi lar for arrays, slices, maps, and str ucts. The specific for m ab ove is a list of values in order, but it is als o possible to specif y a list of index and value pairs, li ke this:

type Currency int

const (

USD Currency = iota

EUR

GBP

RMB

)

symbol := [...]string{USD: "$", EUR: "9", GBP: "!", RMB: """}

fmt.Println(RMB, symbol[RMB]) // "3 ""

In this for m, indices can appear in any order and som e may be omitt ed; as before, uns pecified values take on the zero value for the element typ e. For ins tance, r := [...]int{99: -1}

defines an array r with 100 elements, all zero except for the last, which has value −1.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.1. ARRAYS

83

If an array’s element typ e is comp arab le then the array typ e is comparable too, so we may direc tly compare two arrays of that typ e using the == op erator, which rep orts whether all corresp onding elements are equ al. The != op erator is its negat ion.

a := [2]int{1, 2}

b := [...]int{1, 2}

c := [2]int{1, 3}

fmt.Println(a == b, a == c, b == c) // "true false false"

d := [3]int{1, 2}

fmt.Println(a == d) // compile error: cannot compare [2]int == [3]int As a more plausible example, the function Sum256 in the crypto/sha256 package pro duces the SHA256 cryptographic hash or di gest of a message store d in an arbit rar y byte slice. The digest has 256 bits, so its typ e is [32]byte. If two digests are the same, it is ext remely likely that the two messages are the same; if the digests differ, the two messages are dif ferent. This prog ram pr ints and compares the SHA256 digests of "x" and "X": gopl.io/ch4/sha256

import "crypto/sha256"

func main() {

c1 := sha256.Sum256([]byte("x"))

c2 := sha256.Sum256([]byte("X"))

fmt.Printf("%x\n%x\n%t\n%T\n", c1, c2, c1 == c2, c1) ptg16091132

// Output:

// 2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881

// 4b68ab3847feda7d6c62c1fbcbeebfa35eab7351ed5e78f4ddadea5df64b8015

// false

// [32]uint8

}

The two inp uts differ by only a single bit, but approximately half the bits are dif ferent in the digests. Not ice the Printf verbs: %x to print all the elements of an array or slice of bytes in hexade cimal, %t to show a boole an, and %T to displ ay the typ e of a value.

Wh en a function is cal le d, a copy of each argument value is assig ned to the cor responding parameter variable, so the function receives a copy, not the original. Passing large arrays in this way can be inefficient, and any changes that the function makes to array elements affe ct on ly the copy, not the original. In this regard, Go tre ats arrays like any other typ e, but this behavior is dif ferent fro m languages that imp licitly pass arrays by reference.

Of course, we can explicitly pass a point er to an array so that any modification s the function makes to array elements will be visible to the cal ler. This function zeroes the contents of a

[32]byte ar ray:

func zero(ptr *[32]byte) {

for i := range ptr {

ptr[i] = 0

}

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

84

CHAPTER 4. COMPOSITE TYPES

The array literal [32]byte{} yields an array of 32 bytes. Each element of the array has the zero value for byte, which is zero. We can use that fac t to write a dif ferent version of zero: func zero(ptr *[32]byte) {

*ptr = [32]byte{}

}

Using a point er to an array is efficient and allows the cal le d func tion to mut ate the cal ler’s var iable, but arrays are still inherently inflexible because of their fixe d size. The zero func tion wi l l not accept a point er to a [16]byte var iable, for example, nor is there any way to add or remove array elements. For these reasons, other than speci al cases like SHA256’s fixe d-size hash, arrays are seldom used as function parameters; instead, we use slices.

Exercis e 4.1: Wr ite a function that counts the number of bits that are dif ferent in two SHA256

hashes. (See PopCount from Sec tion 2.6.2.)

Exercis e 4.2: Wr ite a program that prints the SHA256 hash of its stand ard inp ut by defau lt but supp orts a command-line flag to print the SHA384 or SHA512 hash ins tead.

4.2. Slices

Slices represent var iable-lengt h sequences whose elements all have the same typ e. A slice typ e is writt en []T, where the elements have typ e T; it looks like an array typ e without a size.

ptg16091132

Ar rays and slices are int imate ly conne cte d. A slice is a lig htweig ht dat a st ruc ture that gives access to a subsequence (or perhaps all) of the elements of an array, which is known as the slice’s un derly ing array. A slice has three components: a point er, a lengt h, and a cap acity. The pointer points to the firs t element of the array that is reach able through the slice, which is not ne cessarily the array’s firs t element. The lengt h is the number of slice elements; it can’t exceed the cap acity, which is usu ally the number of elements bet ween the start of the slice and the end of the underly ing array. The bui lt-in functions len and cap return thos e values.

Mu ltiple slices can share the same underly ing array and may refer to overl apping par ts of that ar ray. Figure 4.1 shows an array of str ings for the mont hs of the year, and two overl apping slices of it. The array is declare d as

months := [...]string{1: "January", /* ... */, 12: "December"}

so January is months[1] and December is months[12]. Ordinar ily, the array element at index 0 wou ld contain the firs t value, but because months are always numbere d from 1, we can leave it out of the declarat ion and it will be initialize d to an emp ty str ing .

The sli ce operator s[i:j], where 0 ≤ i ≤ j ≤ cap(s), cre ates a new slice that refers to elements i thro ugh j-1 of the sequence s, which may be an array variable, a point er to an array, or anot her slice. The resulting slice has j-i elements. If i is omitt ed, it’s 0, and if j is omitt ed, it’s len(s). Thu s the slice months[1:13] refers to the whole range of valid months, as does the slice months[1:]; the slice months[:] refers to the whole array. Let’s define overl apping slices for the secon d qu arter and the northern summer :

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.2. SLICES

85

ptg16091132

Figure 4.1. Tw o ov erl apping slices of an array of months.

Q2 := months[4:7]

summer := months[6:9]

fmt.Println(Q2)

// ["April" "May" "June"]

fmt.Println(summer) // ["June" "July" "August"]

June is include d in each and is the sole out put of this (inefficient) test for common elements: for _, s := range summer {

for _, q := range Q2 {

if s == q {

fmt.Printf("%s appears in both\n", s)

}

}

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

86

CHAPTER 4. COMPOSITE TYPES

Slicing beyon d cap(s) caus es a panic, but slicing beyon d len(s) extends the slice, so the resu lt may be lon g er than the original:

fmt.Println(summer[:20]) // panic: out of range

endlessSummer := summer[:5] // extend a slice (within capacity) fmt.Println(endlessSummer) // "[June July August September October]"

As an aside, not e the simi lar ity of the subst ring operat ion on str ings to the slice operator on

[]byte slices. Bot h are writt en x[m:n], and bot h return a subsequence of the original bytes, sh aring the underly ing represent ation so that bot h op erat ions take con stant time. The expression x[m:n] yields a str ing if x is a str ing , or a []byte if x is a []byte.

Since a slice cont ains a point er to an element of an array, passing a slice to a function per mits the function to modif y the underly ing array elements. In other words, copying a slice creates an alias (§2.3.2) for the underly ing array. The function reverse re verses the elements of an

[]int slice in place, and it may be app lie d to slices of any lengt h.

gopl.io/ch4/rev

// reverse reverses a slice of ints in place.

func reverse(s []int) {

for i, j := 0, len(s)-1; i < j; i, j = i+1, j-1 {

s[i], s[j] = s[j], s[i]

}

}

ptg16091132

Here we reverse the whole array a:

a := [...]int{0, 1, 2, 3, 4, 5}

reverse(a[:])

fmt.Println(a) // "[5 4 3 2 1 0]"

A simple way to ro tat e a slice left by n elements is to app l y the reverse func tion three times, firs t to the leading n elements, then to the remaining elements, and finally to the whole slice.

(To rot ate to the rig ht, make the third cal l firs t.) s := []int{0, 1, 2, 3, 4, 5}

// Rotate s left by two positions.

reverse(s[:2])

reverse(s[2:])

reverse(s)

fmt.Println(s) // "[2 3 4 5 0 1]"

No tice how the expression that initializes the slice s dif fers fro m that for the array a. A sli ce litera l lo oks like an array literal, a sequence of values sep arated by commas and sur rounded by braces, but the size is not given. This imp licitly cre ates an array variable of the rig ht size and yields a slice that points to it. As wit h ar ray literals, slice literals may specif y the values in order, or give their indices explicitly, or use a mix of the two sty les.

Un like arrays, slices are not comparable, so we cannot use == to test whether two slices cont ain the same elements. The stand ard librar y prov ides the hig hly opt imize d bytes.Equal func tion for comparing two slices of bytes ([]byte), but for other typ es of slice, we must do the

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.2. SLICES

87

comp arison ourselves:

func equal(x, y []string) bool {

if len(x) != len(y) {

return false

}

for i := range x {

if x[i] != y[i] {

return false

}

}

return true

}

Gi ven how natural this ‘‘de ep’’ equality test is, and that it is no more costly at run time than the

== op erator for arrays of str ings, it may be puzzling that slice comp arisons do not als o work this way. There are two reasons why deep equivalence is pro blemat ic. First, unlike array elements, the elements of a slice are indirec t, mak ing it possible for a slice to contain its elf.

Although there are ways to deal wit h such cas es, none is simple, efficient, and most importantly, obv iou s.

Second, because slice elements are indirec t, a fixe d slice value may cont ain different elements at different times as the contents of the underly ing array are modified. Because a hash table such as Go’s map typ e makes only shallow copies of its keys, it requires that equ ality for each ptg16091132

ke y remain the same throughout the lifet ime of the hash table. Deep equivalence wou ld thu s make slices unsuit able for use as map keys. For reference typ es li ke point ers and channel s, the

== op erator tests reference ident ity, that is, whether the two ent ities refer to the same thing. An analogou s ‘‘sh allow’’ equality test for slices could be useful, and it wou ld solve the pro blem with maps, but the incon sistent tre atment of slices and arrays by the == op erator wou ld be conf using. The safest choice is to dis allow slice comp arisons alt ogether.

The only legal slice comp arison is against nil, as in

if summer == nil { /* ... */ }

The zero value of a slice typ e is nil. A ni l slice has no underly ing array. The nil slice has lengt h and cap acity zero, but there are als o non-ni l slices of lengt h and cap acity zero, such as

[]int{} or make([]int, 3)[3:]. As wit h any typ e that can have nil values, the nil value of a partic ular slice typ e can be writt en using a conversion expression such as []int(nil).

var s []int

// len(s) == 0, s == nil

s = nil

// len(s) == 0, s == nil

s = []int(nil) // len(s) == 0, s == nil

s = []int{}

// len(s) == 0, s != nil

So, if you need to test whether a slice is emp ty, use len(s) == 0, not s == nil. Other than comp aring equ al to nil, a nil slice beh aves like any other zero-lengt h slice; reverse(nil) is perfec tly safe, for example. Unless cle arly documente d to the contrar y, Go functions should treat all zero-lengt h slices the same way, whether nil or non-ni l.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

88

CHAPTER 4. COMPOSITE TYPES

The bui lt-in function make creates a slice of a specified element typ e, lengt h, and cap acity. The capacity argument may be omitt ed, in which cas e the cap acity equ als the lengt h.

make([]T, len)

make([]T, len, cap) // same as make([]T, cap)[:len]

Un der the hood, make creates an unnamed array variable and retur ns a slice of it; the array is accessible only through the retur ned slice. In the firs t form, the slice is a vie w of the ent ire ar ray. In the secon d, the slice is a vie w of only the array’s firs t len elements, but its cap acity includes the ent ire array. The addition al elements are set aside for fut ure growt h.

4.2.1. The append Function

The bui lt-in append func tion app end s it ems to slices: var runes []rune

for _, r := range "Hello, BF" {

runes = append(runes, r)

}

fmt.Printf("%q\n", runes) // "['H' 'e' 'l' 'l' 'o' ',' ' ' 'B' 'F']"

The loop uses append to bui ld the slice of nine runes enco ded by the str ing lit eral, alt hough ptg16091132

this specific pro blem is more conveniently solved by using the bui lt-in conv ersion

[]rune("Hello, BF").

The append func tion is cruci al to underst anding how slices wor k, so let’s take a look at what is going on. Here’s a version cal le d appendInt that is speci alized for []int slices: gopl.io/ch4/append

func appendInt(x []int, y int) []int {

var z []int

zlen := len(x) + 1

if zlen <= cap(x) {

// There is room to grow.

Extend the slice.

z = x[:zlen]

} else {

// There is insufficient space.

Allocate a new array.

// Grow by doubling, for amortized linear complexity.

zcap := zlen

if zcap < 2*len(x) {

zcap = 2 * len(x)

}

z = make([]int, zlen, zcap)

copy(z, x) // a built-in function; see text

}

z[len(x)] = y

return z

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.2. SLICES

89

Each cal l to appendInt mu st che ck whether the slice has sufficient cap acity to hold the new elements in the exist ing array. If so, it extends the slice by defining a larger slice (still wit hin the original array), copies the element y into the new space, and retur ns the slice. The inp ut x and the result z sh are the same underly ing array.

If there is ins ufficient space for growt h, appendInt mu st allocate a new array big enough to hold the result, copy the values fro m x into it, then app end the new element y. The result z now refers to a dif ferent underly ing array than the array that x refers to.

It wou ld be straig htfor ward to copy the elements wit h explicit loops, but it’s easier to use the built-in function copy, which copies elements fro m on e slice to another of the same typ e. Its firs t argument is the destination and its secon d is the source, res embling the order of operands in an assig nment like dst = src. The slices may refer to the same underly ing array ; they may even overl ap. Alt hough we don’t use it here, copy returns the number of elements actu ally copied, which is the smaller of the two slice lengt hs, so there is no danger of running off the end or overwriting som ething out of range .

Fo r efficiency, the new array is usu ally som ewhat larger than the minimum needed to hold x and y. Exp anding the array by dou bling its size at each exp ansion avoids an excessive number of allocat ions and ens ures that app ending a single element takes cons tant time on average.

This program demon strates the effec t:

ptg16091132

func main() {

var x, y []int

for i := 0; i < 10; i++ {

y = appendInt(x, i)

fmt.Printf("%d cap=%d\t%v\n", i, cap(y), y)

x = y

}

}

Each change in cap acity indic ates an allo cat ion and a copy: 0

cap=1

[0]

1

cap=2

[0 1]

2

cap=4

[0 1 2]

3

cap=4

[0 1 2 3]

4

cap=8

[0 1 2 3 4]

5

cap=8

[0 1 2 3 4 5]

6

cap=8

[0 1 2 3 4 5 6]

7

cap=8

[0 1 2 3 4 5 6 7]

8

cap=16 [0 1 2 3 4 5 6 7 8]

9

cap=16 [0 1 2 3 4 5 6 7 8 9]

Let’s take a clos er lo ok at the i=3 it erat ion. The slice x cont ains the three elements [0 1 2] but has cap acity 4, so there is a single element of slack at the end, and appendInt of the element 3

may pro ceed wit hout reallocat ing. The resulting slice y has lengt h and cap acity 4, and has the same underly ing array as the original slice x, as Figure 4.2 shows.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

90

CHAPTER 4. COMPOSITE TYPES

Figure 4.2. Ap pending wit h ro om to grow.

On the next iterat ion, i=4, there is no slack at all, so appendInt al locates a new array of size 8, copies the four elements [0 1 2 3] of x, and app end s 4, the value of i. The resulting slice y has a lengt h of 5 but a cap acity of 8; the slack of 3 will save the next three iterat ions fro m the ne e d to reallocate. The slices y and x are vie ws of dif ferent arrays. This operat ion is depic ted in Figure 4.3.

ptg16091132

Figure 4.3. Ap pending wit hout room to grow.

The bui lt-in append func tion may use a more sop histicate d growth strateg y than appendInt’s simplistic one. Usu ally we don’t know whether a given cal l to append wi l l caus e a reallocat ion, so we can’t assume that the original slice refers to the same array as the resulting slice, nor that it refers to a dif ferent one. Simi larly, we must not assume that operat ions on elements of the old slice will (or will not) be reflec ted in the new slice. As a result, it’s usu al to assig n the result of a cal l to append to the same slice variable whose value we passed to append: runes = append(runes, r)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.2. SLICES

91

Up dat ing the slice variable is required not just when cal ling append, but for any function that may change the lengt h or cap acity of a slice or make it refer to a dif ferent underly ing array. To us e slices correc tly, it’s imp ortant to bear in mind that alt hough the elements of the underly ing ar ray are indirec t, the slice’s point er, lengt h, and cap acity are not. To update them requires an assig nment like the one above . In this respect, slices are not ‘‘pure’’ reference typ es but res emble an aggregate typ e such as this str uct:

type IntSlice struct {

ptr

*int

len, cap int

}

Our appendInt func tion adds a single element to a slice, but the bui lt-in append lets us add more than one new element, or even a whole slice of them.

var x []int

x = append(x, 1)

x = append(x, 2, 3)

x = append(x, 4, 5, 6)

x = append(x, x...) // append the slice x

fmt.Println(x)

// "[1 2 3 4 5 6 1 2 3 4 5 6]"

Wi th the small modification shown below, we can match the beh avior of the bui lt-in append.

The ellipsis ‘‘...’’ in the declarat ion of appendInt makes the function variadic: it accepts any ptg16091132

numb er of final arguments. The cor responding ellipsis in the cal l ab ove to append shows how to sup ply a list of arguments fro m a slice. We’ll explain this mechanism in detai l in Section 5.7.

func appendInt(x []int, y ...int) []int {

var z []int

zlen := len(x) + len(y)

// ...expand z to at least zlen...

copy(z[len(x):], y)

return z

}

The log ic to exp and z’s underly ing array remains unchange d and is not shown.

4.2.2. In-Place Slice Techniques

Let’s see more examples of functions that, like rotate and reverse, modif y the elements of a slice in place. Given a list of str ings, the nonempty func tion retur ns the non-empt y on es: gopl.io/ch4/nonempty

// Nonempty is an example of an in-place slice algorithm.

package main

import "fmt"

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

92

CHAPTER 4. COMPOSITE TYPES

// nonempty returns a slice holding only the non-empty strings.

// The underlying array is modified during the call.

func nonempty(strings []string) []string {

i := 0

for _, s := range strings {

if s != "" {

strings[i] = s

i++

}

}

return strings[:i]

}

The subtle par t is that the inp ut slice and the out put slice share the same underly ing array.

This avoids the need to allocate another array, thoug h of cours e the contents of data are par tly ov erwritt en, as evidence d by the secon d pr int statement: data := []string{"one", "", "three"}

fmt.Printf("%q\n", nonempty(data)) // `["one" "three"]`

fmt.Printf("%q\n", data)

// `["one" "three" "three"]`

Thus we wou ld usu ally write: data = nonempty(data).

The nonempty func tion can also be writt en using append: ptg16091132

func nonempty2(strings []string) []string {

out := strings[:0] // zero-length slice of original

for _, s := range strings {

if s != "" {

out = append(out, s)

}

}

return out

}

Whiche ver variant we use, reusing an array in this way requires that at most one out put value is pro duce d for each inp ut value, which is true of many algor it hms that filt er out elements of a sequence or com bine adj acent ones. Such int ric ate slice usage is the exception, not the rule, but it can be cle ar, efficient, and useful on occ asion.

A slice can be used to imp lement a stack. Given an initial ly emp ty slice stack, we can push a ne w value ont o the end of the slice wit h append:

stack = append(stack, v) // push v

The top of the stack is the last element:

top := stack[len(stack)-1] // top of stack

and shr inking the stack by popping that element is

stack = stack[:len(stack)-1] // pop

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.3. MAPS

93

To remov e an element fro m the midd le of a slice, preserving the order of the remaining elements, use copy to slide the hig her-numbere d elements down by one to fill the gap: func remove(slice []int, i int) []int {

copy(slice[i:], slice[i+1:])

return slice[:len(slice)-1]

}

func main() {

s := []int{5, 6, 7, 8, 9}

fmt.Println(remove(s, 2)) // "[5 6 8 9]"

}

And if we don’t need to preserve the order, we can just mov e the last element int o the gap: func remove(slice []int, i int) []int {

slice[i] = slice[len(slice)-1]

return slice[:len(slice)-1]

}

func main() {

s := []int{5, 6, 7, 8, 9}

fmt.Println(remove(s, 2)) // "[5 6 9 8]

}

Exercis e 4.3: Re writ e reverse to use an array point er ins tead of a slice.

ptg16091132

Exercis e 4.4: Wr ite a version of rotate that operates in a single pass.

Exercis e 4.5: Wr ite an in-place function to eliminate adj acent dup lic ates in a []string slice.

Exercis e 4.6: Wr ite an in-place function that squ ashes each run of adj acent Unico de sp aces (s ee unicode.IsSpace) in a UTF-8-enco ded []byte slice into a single ASCII space.

Exercis e 4.7: Mo dif y reverse to reverse the charac ters of a []byte slice that represents a UTF-8-enco ded str ing , in place. Can you do it wit hout allocat ing new memor y?

4.3. Maps

The hash table is one of the most ingenious and versatile of all dat a st ruc tures. It is an unordered col lec tion of key/value pairs in which all the keys are distinc t, and the value asso ciated wit h a given key can be ret rie ved, updated, or remov ed using a con stant numb er of key comp arisons on the average, no matter how large the hash table.

In Go, a map is a reference to a hash table, and a map typ e is writt en map[K]V, where K and V

are the typ es of its keys and values. All of the keys in a given map are of the same typ e, and all of the values are of the same typ e, but the keys need not be of the same typ e as the values. The ke y type K mu st be comparable using ==, so that the map can test whether a given key is equ al to one already wit hin it. Though floating-p oint numbers are comparable, it’s a bad idea to comp are floats for equ ality and, as we mention ed in Chapt er 3, esp eci ally bad if NaN is a possible value. There are no restr ictions on the value typ e V.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

94

CHAPTER 4. COMPOSITE TYPES

The bui lt-in function make can be used to cre ate a map: ages := make(map[string]int) // mapping from strings to ints We can also use a map lit era l to cre ate a new map popu lated wit h some initial key/value pairs: ages := map[string]int{

"alice":

31,

"charlie": 34,

}

This is equivalent to

ages := make(map[string]int)

ages["alice"] = 31

ages["charlie"] = 34

so an alt ernat ive expression for a new emp ty map is map[string]int{}.

Map elements are accessed through the usu al subscript not ation: ages["alice"] = 32

fmt.Println(ages["alice"]) // "32"

and remov ed wit h the bui lt-in function delete:

delete(ages, "alice") // remove element ages["alice"]

ptg16091132

Al l of these operat ions are safe even if the element isn’t in the map; a map lookup using a key that isn’t present retur ns the zero value for its typ e, so, for ins tance, the fol low ing works even when "bob" is not yet a key in the map because the value of ages["bob"] wi l l be 0.

ages["bob"] = ages["bob"] + 1 // happy birthday!

The shorthand assig nment for ms x += y and x++ also wor k for map elements, so we can re-wr ite the statement above as

ages["bob"] += 1

or even more con cis ely as

ages["bob"]++

But a map element is not a var iable, and we cannot take its address: _ = &ages["bob"] // compile error: cannot take address of map element One reason that we can’t take the address of a map element is that growing a map might cause re hashing of exist ing elements into new storage locat ions, thu s potent ial ly invalid ating the address.

To enumerate all the key/value pairs in the map, we use a range-b ased for lo op simi lar to thos e we saw for slices. Successive iterat ions of the loop cause the name and age var iables to be set to the next key/value pair :

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.3. MAPS

95

for name, age := range ages {

fmt.Printf("%s\t%d\n", name, age)

}

The order of map iterat ion is uns pecified, and dif ferent imp lementation s mig ht use a dif ferent hash function, leading to a dif ferent order ing. In prac tice, the order is random, varying fro m on e exec ution to the next. This is int ent ion al; mak ing the sequence vary helps force programs to be robust across imp lementation s. To enumerate the key/value pairs in order, we must sor t the keys explicitly, for ins tance, using the Strings func tion fro m the sort package if the keys are str ings. This is a common pattern:

import "sort"

var names []string

for name := range ages {

names = append(names, name)

}

sort.Strings(names)

for _, name := range names {

fmt.Printf("%s\t%d\n", name, ages[name])

}

Since we know the final size of names from the outset, it is more efficient to allocate an array of the required size up fro nt. The statement below cre ates a slice that is initial ly emp ty but has sufficient cap acity to hold all the keys of the ages map: ptg16091132

names := make([]string, 0, len(ages))

In the firs t range lo op above , we require only the keys of the ages map, so we omit the secon d lo op var iable. In the secon d lo op, we require only the elements of the names slice, so we use the blank identifier _ to ignore the firs t var iable, the index.

The zero value for a map typ e is nil, that is, a reference to no hash table at all.

var ages map[string]int

fmt.Println(ages == nil)

// "true"

fmt.Println(len(ages) == 0) // "true"

Most operat ions on maps, including looku p, delete, len, and range lo ops, are safe to perform on a nil map reference, since it beh aves like an emp ty map. But storing to a nil map caus es a panic:

ages["carol"] = 21 // panic: assignment to entry in nil map Yo u mu st allocate the map before you can store int o it.

Accessing a map element by subscript ing always yields a value. If the key is present in the map, you get the cor responding value; if not, you get the zero value for the element typ e, as we saw wit h ages["bob"]. For many pur pos es that’s fine, but som etimes you need to know whet her the element was really there or not. For example, if the element typ e is numer ic, you mig ht have to distinguish bet ween a non existent element and an element that happens to have the value zero, using a test like this:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

96

CHAPTER 4. COMPOSITE TYPES

age, ok := ages["bob"]

if !ok { /* "bob" is not a key in this map; age == 0. */ }

Yo u’ll often see these two statements com bine d, like this: if age, ok := ages["bob"]; !ok { /* ... */ }

Su bscript ing a map in this context yields two values; the secon d is a boole an that rep orts whet her the element was present. The boole an var iable is often cal le d ok, esp eci ally if it is immediate ly used in an if condit ion.

As wit h slices, maps cannot be compare d to each other ; the only legal comp arison is wit h nil.

To test whether two maps cont ain the same keys and the same ass oci ated values, we must wr ite a loop:

func equal(x, y map[string]int) bool {

if len(x) != len(y) {

return false

}

for k, xv := range x {

if yv, ok := y[k]; !ok || yv != xv {

return false

}

}

ptg16091132

return true

}

Observe how we use !ok to distinguish the ‘‘missing’’ and ‘‘pres ent but zero’’ cases. Had we naïvely writt en xv != y[k], the cal l below wou ld incor rec tly rep ort its arguments as equ al:

// True if equal is written incorrectly.

equal(map[string]int{"A": 0}, map[string]int{"B": 42}) Go does not provide a set type, but since the keys of a map are distinc t, a map can ser ve this purpos e. To illustrate, the program dedup re ads a sequence of lines and prints only the firs t occurrence of each distinc t line. (It’s a var iant of the dup prog ram that we showe d in Section 1.3.) The dedup prog ram us es a map whose keys represent the set of lines that have already app eared to ens ure that subsequent occ urrences are not print ed.

gopl.io/ch4/dedup

func main() {

seen := make(map[string]bool) // a set of strings

input := bufio.NewScanner(os.Stdin)

for input.Scan() {

line := input.Text()

if !seen[line] {

seen[line] = true

fmt.Println(line)

}

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.3. MAPS

97

if err := input.Err(); err != nil {

fmt.Fprintf(os.Stderr, "dedup: %v\n", err)

os.Exit(1)

}

}

Go programmers often descr ibe a map used in this fashion as a ‘‘set of str ings’’ without fur ther ado, but beware, not all map[string]bool values are simple sets; som e may contain bot h true and false values.

Sometimes we need a map or set whose keys are slices, but because a map’s keys must be comparable, this cannot be express ed direc tly. How ever, it can be don e in two steps. First we define a helper function k that maps each key to a str ing , with the pro per ty that k(x) == k(y) if and only if we con sider x and y equivalent. Then we cre ate a map whose keys are str ings, apply ing the helper function to each key before we access the map.

The example below uses a map to record the number of times Add has been cal le d with a given list of str ings. It uses fmt.Sprintf to convert a slice of str ings int o a single str ing that is a suit able map key, quoting each slice element wit h %q to record str ing bound aries faithfully: var m = make(map[string]int)

func k(list []string) string { return fmt.Sprintf("%q", list) }

ptg16091132

func Add(list []string)

{ m[k(list)]++ }

func Count(list []string) int { return m[k(list)] }

The same appro ach can be used for any non-comp arable key typ e, not just slices. It’s even us eful for comparable key typ es when you want a definition of equ ality other than ==, such as case-insensit ive comparisons for str ings. And the typ e of k(x) ne e dn’t be a str ing; any comparable typ e with the desired equivalence pro per ty will do, such as int egers, arrays, or str ucts.

Here’s another example of maps in action, a program that counts the occ urrences of each distinc t Unico de co de point in its input. Since there are a large number of possible charac ters, on ly a small frac tion of which wou ld app ear in any par tic ular document, a map is a natural way to keep track of just the ones that have been seen and their corresponding counts.

gopl.io/ch4/charcount

// Charcount computes counts of Unicode characters.

package main

import (

"bufio"

"fmt"

"io"

"os"

"unicode"

"unicode/utf8"

)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

98

CHAPTER 4. COMPOSITE TYPES

func main() {

counts := make(map[rune]int)

// counts of Unicode characters

var utflen [utf8.UTFMax + 1]int // count of lengths of UTF-8 encodings invalid := 0

// count of invalid UTF-8 characters

in := bufio.NewReader(os.Stdin)

for {

r, n, err := in.ReadRune() // returns rune, nbytes, error if err == io.EOF {

break

}

if err != nil {

fmt.Fprintf(os.Stderr, "charcount: %v\n", err) os.Exit(1)

}

if r == unicode.ReplacementChar && n == 1 {

invalid++

continue

}

counts[r]++

utflen[n]++

}

fmt.Printf("rune\tcount\n")

for c, n := range counts {

ptg16091132

fmt.Printf("%q\t%d\n", c, n)

}

fmt.Print("\nlen\tcount\n")

for i, n := range utflen {

if i > 0 {

fmt.Printf("%d\t%d\n", i, n)

}

}

if invalid > 0 {

fmt.Printf("\n%d invalid UTF-8 characters\n", invalid)

}

}

The ReadRune method perfor ms UTF-8 deco ding and retur ns three values: the decoded rune, the lengt h in bytes of its UTF-8 enco ding, and an error value. The only error we exp ect is end-of-file. If the inp ut was not a legal UTF-8 enco ding of a rune, the retur ned rune is unicode.ReplacementChar and the lengt h is 1.

The charcount prog ram also prints a count of the lengt hs of the UTF-8 enco dings of the runes that app eared in the inp ut. A map is not the best dat a st ruc ture for that; since enco ding lengt hs range only fro m 1 to utf8.UTFMax (w hich has the value 4), an array is more compact.

As an exp eriment, we ran charcount on this book its elf at one point. Alt hough it’s mostly in English, of cours e, it does have a fair number of non-ASCII charac ters. Here are the top ten:

°

27

B

15

F

14 é 13

A

10

<

5

&

5

D

4

(

4

+

3

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.4. STRUCTS

99

and here is the distr ibution of the lengt hs of all the UTF-8 enco dings: len count

1

765391

2

60

3

70

4

0

The value typ e of a map can its elf be a composite typ e, such as a map or slice. In the fol low ing co de, the key typ e of graph is string and the value typ e is map[string]bool, represent ing a set of str ings. Con ceptu ally, graph maps a str ing to a set of rel ate d st rings, its successors in a direc ted graph.

gopl.io/ch4/graph

var graph = make(map[string]map[string]bool)

func addEdge(from, to string) {

edges := graph[from]

if edges == nil {

edges = make(map[string]bool)

graph[from] = edges

}

edges[to] = true

}

func hasEdge(from, to string) bool {

ptg16091132

return graph[from][to]

}

The addEdge func tion shows the idiomatic way to popu late a map lazi ly, that is, to initialize each value as its key app ears for the firs t time. The hasEdge func tion shows how the zero value of a missing map entr y is often put to wor k: even if neither from nor to is present, graph[from][to] wi l l always give a meaningf ul resu lt.

Exercis e 4.8: Mo dif y charcount to count letters, dig its, and so on in their Unico de categories, using functions like unicode.IsLetter.

Exercis e 4.9: Wr ite a program wordfreq to rep ort the fre quency of each word in an inp ut text file. Cal l input.Split(bufio.ScanWords) before the firs t call to Scan to bre ak the inp ut int o word s instead of lines.

4.4. Structs

A st ruc t is an aggregate dat a type that gro ups toget her zero or more named values of arbit rar y types as a single ent ity. Each value is cal le d a fiel d. The classic example of a str uct fro m data processing is the emp loy ee record, whose fields are a unique ID, the emp loy ee’s name, address, date of bir th, position, sal ary, manager, and the like. All of these fields are col lec ted int o a single ent ity that can be copie d as a unit, passed to functions and retur ned by them, store d in ar rays, and so on.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

100

CHAPTER 4. COMPOSITE TYPES

Thes e two statements declare a str uct typ e called Employee and a var iable cal le d dilbert that is an ins tance of an Employee:

type Employee struct {

ID

int

Name

string

Address

string

DoB

time.Time

Position string

Salary

int

ManagerID int

}

var dilbert Employee

The indiv idu al fields of dilbert are accessed using dot not ation like dilbert.Name and dilbert.DoB. Because dilbert is a var iable, its fields are var iables too, so we may assig n to a field:

dilbert.Salary -= 5000 // demoted, for writing too few lines of code or take its address and access it through a point er:

position := &dilbert.Position

*position = "Senior " + *position // promoted, for outsourcing to Elbonia ptg16091132

The dot not ation als o works wit h a point er to a str uct: var employeeOfTheMonth *Employee = &dilbert

employeeOfTheMonth.Position += " (proactive team player)"

The last statement is equivalent to

(*employeeOfTheMonth).Position += " (proactive team player)"

Gi ven an emp loy ee’s unique ID, the function EmployeeByID returns a point er to an Employee st ruc t. We can use the dot not ation to access its fields: func EmployeeByID(id int) *Employee { /* ... */ }

fmt.Println(EmployeeByID(dilbert.ManagerID).Position) // "Pointy-haired boss"

id := dilbert.ID

EmployeeByID(id).Salary = 0 // fired for... no real reason The last statement updates the Employee st ruc t that is point ed to by the result of the cal l to EmployeeByID. If the result typ e of EmployeeByID were change d to Employee instead of

*Employee, the assig nment statement wou ld not compi le since its left-hand side wou ld not identify a var iable.

Fields are usu ally writt en one per line, wit h the field’s name pre ceding its typ e, but con sec utive fields of the same typ e may be com bine d, as wit h Name and Address here:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.4. STRUCTS

101

type Employee struct {

ID

int

Name, Address string

DoB

time.Time

Position

string

Salary

int

ManagerID

int

}

Field order is sig nificant to typ e identity. Had we als o combined the declarat ion of the Position field (also a str ing), or int erc hange d Name and Address, we wou ld be defining a dif ferent st ruc t type. Typic ally we only com bine the declarat ions of rel ate d fields.

The name of a str uct field is exp orted if it beg ins wit h a capit al letter; this is Go’s main access cont rol mechanism. A st ruc t type may cont ain a mixture of exp orted and unexp orted fields.

St ruc t types tend to be verb ose because the y of ten invo l ve a line for each field. Alt hough we could write out the whole typ e each time it is needed, the rep etit ion wou ld get tires ome.

In ste ad, str uct typ es usually app ear wit hin the declarat ion of a named typ e li ke Employee.

A named str uct typ e S can’t declare a field of the same typ e S: an aggregate value cannot contain its elf. (An analogou s rest ric tion app lies to arrays.) But S may declare a field of the pointer typ e *S, which lets us cre ate rec ursive dat a st ruc tures like lin ked lists and tre es. This is illustrated in the code below, which uses a binar y tree to imp lement an ins ertion sor t: ptg16091132

gopl.io/ch4/treesort

type tree struct {

value

int

left, right *tree

}

// Sort sorts values in place.

func Sort(values []int) {

var root *tree

for _, v := range values {

root = add(root, v)

}

appendValues(values[:0], root)

}

// appendValues appends the elements of t to values in order

// and returns the resulting slice.

func appendValues(values []int, t *tree) []int {

if t != nil {

values = appendValues(values, t.left)

values = append(values, t.value)

values = appendValues(values, t.right)

}

return values

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

102

CHAPTER 4. COMPOSITE TYPES

func add(t *tree, value int) *tree {

if t == nil {

// Equivalent to return &tree{value: value}.

t = new(tree)

t.value = value

return t

}

if value < t.value {

t.left = add(t.left, value)

} else {

t.right = add(t.right, value)

}

return t

}

The zero value for a str uct is composed of the zero values of each of its fields. It is usu ally desirable that the zero value be a natural or sensible defau lt. For example, in bytes.Buffer, the initial value of the str uct is a ready-to-use emp ty buf fer, and the zero value of sync.Mutex, which we’ll see in Chapt er 9, is a ready-to-use unlocke d mu tex. Som etimes this sensible initial behavior happens for fre e, but som etimes the typ e desig ner has to wor k at it.

The str uct typ e with no fields is cal le d the empty str uct, writt en struct{}. It has size zero and carries no infor mat ion but may be useful non etheless. Som e Go programmers use it ins tead of bool as the value typ e of a map that represents a set, to emp hasize that only the keys are sig-ptg16091132

nificant, but the space sav ing is marginal and the syntax more cumbers ome, so we general ly avoid it.

seen := make(map[string]struct{}) // set of strings

// ...

if _, ok := seen[s]; !ok {

seen[s] = struct{}{}

// ...first time seeing s...

}

4.4.1. Struct Literals

A value of a str uct typ e can be writt en using a st ruc t litera l that specifies values for its fields.

type Point struct{ X, Y int }

p := Point{1, 2}

There are two for ms of str uct lit eral. The firs t form, shown above , re quires that a value be sp ecified for ever y field, in the rig ht order. It burdens the writer (and reader) wit h rememb ering exac tly what the fields are , and it makes the code frag i le should the set of fields later grow or be reordered. Accordingly, this for m tend s to be used only wit hin the package that defines the str uct typ e, or wit h smal ler st ruc t types for which there is an obv iou s field order ing convent ion, like image.Point{x, y} or color.RGBA{red, green, blue, alpha}.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.4. STRUCTS

103

Mo re often, the secon d form is used, in which a str uct value is initialize d by listing som e or all of the field names and their corresponding values, as in this statement fro m the Lissajous prog ram of Sec tion 1.4:

anim := gif.GIF{LoopCount: nframes}

If a field is omitt ed in this kind of lit eral, it is set to the zero value for its typ e. Because names are provide d, the order of fields doesn’t matter.

The two for ms cannot be mixed in the same lit eral. Nor can you use the (order-b ased) firs t form of lit eral to sne ak aro und the rule that unexp orted identifiers may not be refer red to from another package.

package p

type T struct{ a, b int } // a and b are not exported

package q

import "p"

var _ = p.T{a: 1, b: 2} // compile error: can't reference a, b var _ = p.T{1, 2}

// compile error: can't reference a, b

Although the last line above doesn’t mention the unexp orted field identifiers, it’s really using them implicitly, so it’s not allow ed.

St ruc t values can be passed as arguments to functions and retur ned fro m them. For ins tance, ptg16091132

this function scales a Point by a specified fac tor :

func Scale(p Point, factor int) Point {

return Point{p.X * factor, p.Y * factor}

}

fmt.Println(Scale(Point{1, 2}, 5)) // "{5 10}"

Fo r efficiency, larger str uct typ es are usu ally passed to or retur ned fro m func tions indirec tly using a point er,

func Bonus(e *Employee, percent int) int {

return e.Salary * percent / 100

}

and this is required if the function must modif y its argument, since in a cal l-by-value langu age li ke Go, the cal le d func tion receives only a copy of an argument, not a reference to the original argument.

func AwardAnnualRaise(e *Employee) {

e.Salary = e.Salary * 105 / 100

}

Becaus e st ruc ts are so commonly dealt wit h thro ugh point ers, it’s possible to use this shorthand notation to cre ate and initialize a struct var iable and obt ain its address: pp := &Point{1, 2}

It is exac tly equivalent to

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

104

CHAPTER 4. COMPOSITE TYPES

pp := new(Point)

*pp = Point{1, 2}

but &Point{1, 2} can be used direc tly wit hin an expression, such as a function cal l.

4.4.2. Comparing Structs

If all the fields of a str uct are comparable, the str uct its elf is comparable, so two expressions of that typ e may be compare d using == or !=. The == op erat ion compares the cor responding fields of the two str ucts in order, so the two print ed expressions below are equivalent: type Point struct{ X, Y int }

p := Point{1, 2}

q := Point{2, 1}

fmt.Println(p.X == q.X && p.Y == q.Y) // "false"

fmt.Println(p == q)

// "false"

Comp arable str uct typ es, li ke other comp arable typ es, may be used as the key typ e of a map.

type address struct {

hostname string

port

int

ptg16091132

}

hits := make(map[address]int)

hits[address{"golang.org", 443}]++

4.4.3. Struct Embedding and Anonymous Fields

In this sec tion, we’ll see how Go’s unu sual st ruc t embedding me chanism lets us use one named st ruc t type as an an ony mou s fiel d of another str uct typ e, providing a convenient syntactic shortcut so that a simple dot expression like x.f can stand for a chain of fields like x.d.e.f.

Consider a 2-D drawing program that provides a librar y of shapes, such as rec tangles, ellipses, st ars, and whe els. Here are two of the typ es it mig ht define: type Circle struct {

X, Y, Radius int

}

type Wheel struct {

X, Y, Radius, Spokes int

}

A Circle has fields for the X and Y co ordinates of its center, and a Radius. A Wheel has all the fe atures of a Circle, plu s Spokes, the number of ins cribed radi al sp okes. Let’s cre ate a whe el:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.4. STRUCTS

105

var w Wheel

w.X = 8

w.Y = 8

w.Radius = 5

w.Spokes = 20

As the set of shapes grows, we’re bound to not ice similarities and rep etit ion among them, so it may be convenient to fac tor out their common par ts:

type Point struct {

X, Y int

}

type Circle struct {

Center Point

Radius int

}

type Wheel struct {

Circle Circle

Spokes int

}

The app lic ation may be cle arer for it, but this change makes accessing the fields of a Wheel ptg16091132

more verb ose:

var w Wheel

w.Circle.Center.X = 8

w.Circle.Center.Y = 8

w.Circle.Radius = 5

w.Spokes = 20

Go lets us declare a field wit h a typ e but no name; such fields are cal le d an ony mou s fiel ds. The type of the field must be a named typ e or a point er to a named typ e. Below, Circle and Wheel have one anony mou s field each. We say that a Point is embedded within Circle, and a Circle is emb edde d within Wheel.

type Circle struct {

Point

Radius int

}

type Wheel struct {

Circle

Spokes int

}

Than ks to emb edding, we can refer to the names at the leaves of the imp licit tre e without giving the int ervening names:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

106

CHAPTER 4. COMPOSITE TYPES

var w Wheel

w.X = 8

// equivalent to w.Circle.Point.X = 8

w.Y = 8

// equivalent to w.Circle.Point.Y = 8

w.Radius = 5

// equivalent to w.Circle.Radius = 5

w.Spokes = 20

The explicit for ms shown in the comments above are still valid, how ever, showing that

‘‘anonymou s field’’ is som ething of a misnomer. The fields Circle and Point do have names—that of the named typ e—but those names are opt ion al in dot expressions. We may omit any or all of the anony mou s fields when selec ting their subfields.

Unfortunately, there’s no cor responding shorthand for the str uct lit eral synt ax, so neither of thes e wi l l comp ile:

w = Wheel{8, 8, 5, 20}

// compile error: unknown fields

w = Wheel{X: 8, Y: 8, Radius: 5, Spokes: 20} // compile error: unknown fields The str uct lit eral must fol low the shap e of the typ e de clarat ion, so we must use one of the two forms below, which are equivalent to each other :

gopl.io/ch4/embed

w = Wheel{Circle{Point{8, 8}, 5}, 20}

w = Wheel{

Circle: Circle{

Point: Point{X: 8, Y: 8},

ptg16091132

Radius: 5,

},

Spokes: 20, // NOTE: trailing comma necessary here (and at Radius)

}

fmt.Printf("%#v\n", w)

// Output:

// Wheel{Circle:Circle{Point:Point{X:8, Y:8}, Radius:5}, Spokes:20}

w.X = 42

fmt.Printf("%#v\n", w)

// Output:

// Wheel{Circle:Circle{Point:Point{X:42, Y:8}, Radius:5}, Spokes:20}

No tice how the # adverb causes Printf’s %v verb to displ ay values in a for m simi lar to Go syntax. For str uct values, this for m includes the name of each field.

Becaus e ‘‘anonymou s’’ fields do have imp licit names, you can’t have two anony mou s fields of the same typ e since their names wou ld conflic t. And because the name of the field is imp licit ly deter mined by its typ e, so too is the visibilit y of the field. In the examples abov e, the Point and Circle anonymou s fields are exp orted. Had the y been unexp orted (point and circle), we cou ld still use the shorthand for m

w.X = 8 // equivalent to w.circle.point.X = 8

but the explicit lon g form shown in the comment wou ld be forbidden outside the declaring package because circle and point would be inaccessible.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.5. JSON

107

Wh at we’ve seen so far of str uct emb edding is just a spr ink ling of syntactic sugar on the dot notation used to selec t st ruc t fields. Later, we’ll see that anony mou s fields need not be str uct types; any named typ e or point er to a named typ e wi l l do. But why wou ld you want to emb ed a typ e that has no subfields?

The answer has to do wit h methods. The shorthand notation used for selec ting the fields of an embedde d type wor ks for selec ting its met hods as wel l. In effec t, the out er str uct typ e gains not just the fields of the emb edde d type but its met hods too. This mechanism is the main way that comp lex obj e ct beh avior s are composed fro m simpler ones. Comp ositi on is central to objec t-oriente d prog ramming in Go, and we’ll explore it fur ther in Sec tion 6.3.

4.5. JSON

JavaS crip t Objec t No tat ion (JSON) is a stand ard not ation for sending and receiving str uctured infor mat ion. JSON is not the only such not ation. XML (§7.14), ASN.1, and Google’s Pro tocol Buffers ser ve simi lar pur pos es and each has its niche, but because of its simplicity, readabi lit y, and univers al sup por t, JSON is the most widely used.

Go has excellent sup por t for encoding and decoding these for mats, provide d by the stand ard librar y packages encoding/json, encoding/xml, encoding/asn1, and so on, and these packages all have simi lar APIs. This sec tion gives a brief overvie w of the most imp ortant par ts of ptg16091132

the encoding/json package.

JSON is an encoding of JavaS crip t values—st rings, numbers, boole ans, arrays, and obj e cts—as Unico de text. It’s an efficient yet readable represent ation for the basic dat a types of Chapt er 3

and the composite typ es of this chapt er—ar rays, slices, str ucts, and maps.

The basic JSON typ es are numbers (in decimal or scientific not ation), boole ans (true or false), and str ings, which are sequences of Unico de co de points enclos ed in dou ble quotes, with backslash escap es using a simi lar not ation to Go, thoug h JSON’s \Uhhhh numeric escap es denot e UTF-16 codes, not runes.

Thes e basic typ es may be com bine d re cursive ly using JSON arrays and obj e cts. A JSON array is an ordered sequence of values, writt en as a comma-s eparated list enclos ed in squ are brackets; JSON arrays are used to encode Go arrays and slices. A JSON obj e ct is a mapping fro m st rings to values, writt en as a sequence of name:value pairs sep arated by commas and sur-ro unded by braces; JSON obj e cts are used to encode Go maps (wit h st ring keys) and str ucts.

Fo r example:

boolean

true

number

-273.15

string

"She said \"Hello, BF\""

array

["gold", "silver", "bronze"]

object

{"year": 1980,

"event": "archery",

"medals": ["gold", "silver", "bronze"]}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

108

CHAPTER 4. COMPOSITE TYPES

Consider an applic ation that gat hers mov ie revie ws and offers recommend ation s. Its Movie data typ e and a typic al list of values are declare d below. (The str ing lit erals after the Year and Color field declarat ions are fiel d tags; we’ll explain them in a mom ent.) gopl.io/ch4/movie

type Movie struct {

Title string

Year

int

`json:"released"`

Color bool `json:"color,omitempty"Àctors []string

}

var movies = []Movie{

{Title: "Casablanca", Year: 1942, Color: false, Actors: []string{"Humphrey Bogart", "Ingrid Bergman"}},

{Title: "Cool Hand Luke", Year: 1967, Color: true, Actors: []string{"Paul Newman"}},

{Title: "Bullitt", Year: 1968, Color: true,

Actors: []string{"Steve McQueen", "Jacqueline Bisset"}},

// ...

}

Data str uctures like this are an excel lent fit for JSON, and it’s easy to convert in bot h direc tion s. Converting a Go dat a st ruc ture like movies to JSON is cal le d marshaling. Mar-ptg16091132

sh aling is don e by json.Marshal:

data, err := json.Marshal(movies)

if err != nil {

log.Fatalf("JSON marshaling failed: %s", err)

}

fmt.Printf("%s\n", data)

Marshal produces a byte slice cont aining a ver y long str ing wit h no ext raneous white space; we’ve folde d the lines so it fits:

[{"Title":"Casablanca","released":1942,"Actors":["Humphrey Bogart","Ingr id Bergman"]},{"Title":"Cool Hand Luke","released":1967,"color":true,"Ac tors":["Paul Newman"]},{"Title":"Bullitt","released":1968,"color":true,"

Actors":["Steve McQueen","Jacqueline Bisset"]}]

This compact represent ation contains all the infor mat ion but it’s hard to read. For human cons ump tion, a variant cal le d json.MarshalIndent produces neatly indente d output. Two addition al arguments define a prefix for each line of out put and a str ing for each level of indentation:

data, err := json.MarshalIndent(movies, "", "

")

if err != nil {

log.Fatalf("JSON marshaling failed: %s", err)

}

fmt.Printf("%s\n", data)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.5. JSON

109

The code above prints

[

{

"Title": "Casablanca",

"released": 1942,

"Actors": [

"Humphrey Bogart",

"Ingrid Bergman"

]

},

{

"Title": "Cool Hand Luke",

"released": 1967,

"color": true,

"Actors": [

"Paul Newman"

]

},

{

"Title": "Bullitt",

"released": 1968,

"color": true,

"Actors": [

ptg16091132

"Steve McQueen",

"Jacqueline Bisset"

]

}

]

Mars haling uses the Go str uct field names as the field names for the JSON obj e cts (through reflec tion, as we’ll see in Sec tion 12.6). Only exp orted fields are marsh ale d, which is why we ch ose capit alize d names for all the Go field names.

Yo u may have not ice d that the name of the Year field change d to released in the out put, and Color ch ange d to color. That’s because of the fiel d tags. A field tag is a str ing of met adat a asso ciate d at comp ile time wit h the field of a str uct: Year int

`json:"released"`

Color bool `json:"color,omitempty"À field tag may be any lit eral str ing , but it is convent ion ally int erprete d as a space-s eparated list of key:"value" pairs; since the y cont ain double quotation marks, field tags are usu ally wr itt en wit h raw str ing lit erals. The json ke y cont rol s the beh avior of the encoding/json package, and other encoding/... packages fol low this convent ion. The firs t part of the json field tag specifies an alternat ive JSON name for the Go field. Field tags are often used to sp ecif y an idiomatic JSON name like total_count for a Go field named TotalCount. The tag for Color has an addition al opt ion, omitempty, which indic ates that no JSON out put should be pro duce d if the field has the zero value for its typ e (false, here) or is other wis e empt y.

Sure enoug h, the JSON out put for Casabl anca, a black-and-w hite mov ie, has no color field.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

110

CHAPTER 4. COMPOSITE TYPES

The inv ers e op erat ion to marsh aling, decoding JSON and popu lat ing a Go dat a st ruc ture, is called unmarshaling, and it is don e by json.Unmarshal. The code below unmarsh als the JSON mov ie dat a into a slice of str ucts whose only field is Title. By defining suitable Go dat a st ruc tures in this way, we can selec t which par ts of the JSON inp ut to decode and which to discard . Wh en Unmarshal returns, it has filled in the slice wit h the Title infor mat ion; other names in the JSON are ignored.

var titles []struct{ Title string }

if err := json.Unmarshal(data, &titles); err != nil {

log.Fatalf("JSON unmarshaling failed: %s", err)

}

fmt.Println(titles) // "[{Casablanca} {Cool Hand Luke} {Bullitt}]"

Many web ser vices provide a JSON int erface—make a request wit h HT TP and back com es the desired infor mat ion in JSON for mat. To illustrate, let’s quer y the GitHu b issue tracker using its web-s ervice interface. First we’ll define the necessary typ es and con stants: gopl.io/ch4/github

// Package github provides a Go API for the GitHub issue tracker.

// See https://developer.github.com/v3/search/#search-issues.

package github

import "time"

const IssuesURL = "https://api.github.com/search/issues"

ptg16091132

type IssuesSearchResult struct {

TotalCount int `json:"total_count"Ìtems

[]*Issue

}

type Issue struct {

Number

int

HTMLURL

string `json:"html_url"`

Title

string

State

string

User

*User

CreatedAt time.Timèjson:"created_at"`

Body

string

// in Markdown format

}

type User struct {

Login

string

HTMLURL string `json:"html_url"`

}

As before, the names of all the str uct fields must be capit alize d even if their JSON names are not. How ever, the matching pro cess that ass oci ates JSON names wit h Go str uct names dur ing unmarsh aling is cas e-insensit ive , so it’s only necessary to use a field tag when there’s an underscore in the JSON name but not in the Go name. Again, we are being selec tive about which fields to decode; the GitHu b search respons e cont ains con siderably more infor mat ion than we show here.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.5. JSON

111

The SearchIssues func tion makes an HTTP request and decodes the result as JSON. Since the quer y terms present ed by a user cou ld contain charac ters like ? and & that have speci al me aning in a URL, we use url.QueryEscape to ens ure that the y are taken literal ly.

gopl.io/ch4/github

package github

import (

"encoding/json"

"fmt"

"net/http"

"net/url"

"strings"

)

// SearchIssues queries the GitHub issue tracker.

func SearchIssues(terms []string) (*IssuesSearchResult, error) {

q := url.QueryEscape(strings.Join(terms, " ")) resp, err := http.Get(IssuesURL + "?q=" + q)

if err != nil {

return nil, err

}

// We must close resp.Body on all execution paths.

ptg16091132

// (Chapter 5 presents 'defer', which makes this simpler.) if resp.StatusCode != http.StatusOK {

resp.Body.Close()

return nil, fmt.Errorf("search query failed: %s", resp.Status)

}

var result IssuesSearchResult

if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {

resp.Body.Close()

return nil, err

}

resp.Body.Close()

return &result, nil

}

The earlier examples used json.Unmarshal to decode the ent ire contents of a byte slice as a single JSON ent ity. For var iety, this example uses the st reami n g de coder, json.Decoder, which allows several JSON ent ities to be decoded in sequence fro m the same stream, although we don’t need that feature here. As you mig ht exp ect, there is a cor responding streaming enco der called json.Encoder.

The cal l to Decode popu lates the var iable result. There are var ious ways we can for mat its value nicely. The simplest, demon strated by the issues command below, is as a text table with fixe d-w idth columns, but in the next sec tion we’ll see a more sop histicate d approach based on templ ates.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

112

CHAPTER 4. COMPOSITE TYPES

gopl.io/ch4/issues

// Issues prints a table of GitHub issues matching the search terms.

package main

import (

"fmt"

"log"

"os"

"gopl.io/ch4/github"

)

func main() {

result, err := github.SearchIssues(os.Args[1:])

if err != nil {

log.Fatal(err)

}

fmt.Printf("%d issues:\n", result.TotalCount) for _, item := range result.Items {

fmt.Printf("#%-5d %9.9s %.55s\n",

item.Number, item.User.Login, item.Title)

}

}

The command-line arguments specif y the searc h terms. The command below quer ies the Go ptg16091132

proj e ct’s issue tracker for the list of open bugs rel ate d to JSON decoding: $ go build gopl.io/ch4/issues

$ ./issues repo:golang/go is:open json decoder

13 issues:

#5680

eaigner encoding/json: set key converter on en/decoder

#6050 gopherbot encoding/json: provide tokenizer

#8658 gopherbot encoding/json: use bufio

#8462 kortschak encoding/json: UnmarshalText confuses json.Unmarshal

#5901

rsc encoding/json: allow override type marshaling

#9812 klauspost encoding/json: string tag not symmetric

#7872 extempora encoding/json: Encoder internally buffers full output

#9650

cespare encoding/json: Decoding gives errPhase when unmarshalin

#6716 gopherbot encoding/json: include field name in unmarshal error me

#6901 lukescott encoding/json, encoding/xml: option to treat unknown fi

#6384

joeshaw encoding/json: encode precise floating point integers u

#6647

btracey x/tools/cmd/godoc: display type kind of each named type

#4237 gjemiller encoding/base64: URLEncoding padding is optional The GitHu b we b-s ervice interface at https://developer.github.com/v3/ has many more fe atures than we have space for here.

Exercis e 4.10: Mo dif y issues to rep ort the results in age categor ies, say less than a month old, less than a year old, and more than a year old.

Exercis e 4.11: Build a tool that lets users cre ate, read, update, and delete GitHu b issues fro m the command line, inv oking their preferred text editor when subst ant ial text input is required.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.6. TEXT AND HTML TEMPLATES

113

Exercis e 4.12: The popu lar web comic xkcd has a JSON int erface. For example, a request to

https://xkcd.com/571/info.0.json produces a detai le d des crip tion of comic 571, one of many favor ites. Dow nlo ad each URL (once!) and bui ld an offline index. Write a tool xkcd that, using this index, prints the URL and transcript of each comic that match es a searc h term prov ide d on the command line.

Exercis e 4.13: The JSON-b ased web ser vice of the Open Mov ie Dat abas e lets you searc h

https://omdbapi.com/ for a mov ie by name and dow nlo ad its poster image . Wr ite a tool poster that down loads the poster image for the movie named on the command line.

4.6. Text and HTML Templates

The pre vious example does only the simplest possible for matting , for which Printf is ent ire ly ade quate. But som etimes for matting must be more elaborate, and it’s desirable to sep arate the format fro m the code more completely. This can be don e with the text/template and html/template packages, which provide a mechanism for subst ituting the values of var iables into a text or HTML templ ate.

A templ ate is a str ing or file cont aining one or more por tions enclos ed in dou ble braces,

{{...}}, cal le d ac tions. Most of the str ing is print ed lit eral ly, but the actions trigger other behavior s. Each action contains an expression in the templ ate langu age, a simple but pow erful ptg16091132

notation for printing values, selec ting str uct fields, cal ling functions and methods, expressing cont rol flow such as if-else st atements and range lo ops, and ins tantiat ing other templ ates.

A simple templ ate str ing is shown below :

gopl.io/ch4/issuesreport

const templ = `{{.TotalCount}} issues:

{{range .Items}}----------------------------------------

Number: {{.Number}}

User:

{{.User.Login}}

Title: {{.Title | printf "%.64s"}}

Age:

{{.CreatedAt | daysAgo}} days

{{end}}`

This templ ate firs t pr ints the number of matching issues, then prints the number, user, tit le, and age in days of each one. Wit hin an action, there is a not ion of the cur rent value, refer red to as ‘‘dot’’ and writt en as ‘‘.’’, a per iod. The dot initial ly refers to the templ ate’s parameter, which will be a github.IssuesSearchResult in this example. The {{.TotalCount}} ac tion exp ands to the value of the TotalCount field, print ed in the usu al way.

The

{{range .Items}} and {{end}} ac tions cre ate a loop, so the text bet ween them is exp ande d mu ltiple times, wit h dot bound to successive elements of Items.

Wi thin an action, the | notation makes the result of one operat ion the argument of another, analogou s to a Unix shel l pip eline. In the cas e of Title, the secon d op erat ion is the printf func tion, which is a bui lt-in synony m for fmt.Sprintf in all templ ates. For Age, the secon d op erat ion is the fol low ing function, daysAgo, which converts the CreatedAt field int o an

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

114

CHAPTER 4. COMPOSITE TYPES

el aps ed time, using time.Since:

func daysAgo(t time.Time) int {

return int(time.Since(t).Hours() / 24)

}

No tice that the typ e of CreatedAt is time.Time, not string. In the same way that a typ e may cont rol its str ing for matting (§2.5) by defining cer tain met hods, a typ e may als o define methods to control its JSON marsh aling and unmarsh aling beh avior. The JSON-marsh ale d value of a time.Time is a str ing in a stand ard for mat.

Producing out put wit h a templ ate is a two-step pro cess. First we must parse the templ ate int o a suitable int ernal represent ation, and then exe cut e it on specific inp uts. Parsing need be don e on ly once. The code below cre ates and parses the templ ate templ define d ab ove . No te the ch aining of met hod cal ls: template.New creates and retur ns a templ ate; Funcs adds daysAgo to the set of functions accessible wit hin this templ ate, then retur ns that templ ate; finally, Parse is cal le d on the result.

report, err := template.New("report").

Funcs(template.FuncMap{"daysAgo": daysAgo}).

Parse(templ)

if err != nil {

log.Fatal(err)

ptg16091132

}

Becaus e temp lates are usu ally fixe d at comp ile time, fai lure to parse a templ ate indic ates a fatal bug in the program. The template.Must helper function makes erro r hand ling more convenient: it accepts a templ ate and an error, che cks that the error is nil (and panics other wis e), and then retur ns the templ ate. We’ll com e back to this ide a in Sec tion 5.9.

Once the templ ate has been cre ate d, aug mente d with daysAgo, parsed, and che cke d, we can exec ute it using a github.IssuesSearchResult as the dat a source and os.Stdout as the destination:

var report = template.Must(template.New("issuelist").

Funcs(template.FuncMap{"daysAgo": daysAgo}).

Parse(templ))

func main() {

result, err := github.SearchIssues(os.Args[1:])

if err != nil {

log.Fatal(err)

}

if err := report.Execute(os.Stdout, result); err != nil {

log.Fatal(err)

}

}

The program prints a plain text rep ort like this:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.6. TEXT AND HTML TEMPLATES

115

$ go build gopl.io/ch4/issuesreport

$ ./issuesreport repo:golang/go is:open json decoder

13 issues:

----------------------------------------

Number: 5680

User:

eaigner

Title: encoding/json: set key converter on en/decoder

Age:

750 days

----------------------------------------

Number: 6050

User:

gopherbot

Title: encoding/json: provide tokenizer

Age:

695 days

----------------------------------------

...

No w let’s tur n to the html/template package. It uses the same API and expression langu age as text/template but adds features for aut omat ic and context-appropriate escaping of str ings appear ing wit hin HTML, JavaS crip t, CSS, or URLs. Thes e fe atures can help avoid a perenni al security pro blem of HTML generat ion, an injection attack, in which an adversary craf ts a st ring value like the tit le of an issue to include malicious code that, when imp rop erly escap ed by a templ ate, gives them control over the page.

ptg16091132

The templ ate below prints the list of issues as an HTML table. Not e the dif ferent imp ort: gopl.io/ch4/issueshtml

import "html/template"

var issueList = template.Must(template.New("issuelist").Parse(`

<h1>{{.TotalCount}} issues</h1>

<table>

<tr style='text-align: left'>

<th>#</th>

<th>State</th>

<th>User</th>

<th>Title</th>

</tr>

{{range .Items}}

<tr>

<td><a href='{{.HTMLURL}}'>{{.Number}}</td>

<td>{{.State}}</td>

<td><a href='{{.User.HTMLURL}}'>{{.User.Login}}</a></td>

<td><a href='{{.HTMLURL}}'>{{.Title}}</a></td>

</tr>

{{end}}

</table>

`))

The command below exe cut es the new templ ate on the results of a slig htly dif ferent quer y:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

116

CHAPTER 4. COMPOSITE TYPES

$ go build gopl.io/ch4/issueshtml

$ ./issueshtml repo:golang/go commenter:gopherbot json encoder >issues.html Figure 4.4 shows the app earance of the table in a web brows er. The lin ks connec t to the appropriate web pages at GitHu b.

ptg16091132

Figure 4.4. An HTML table of Go pro jec t issues rel ating to JSON encoding.

No ne of the issues in Figure 4.4 pos e a challenge for HTML, but we can see the effec t more clearly wit h issues whose tit les cont ain HTML met acharac ters like & and <. We’ve selec ted two such issues for this example:

$ ./issueshtml repo:golang/go 3133 10535 >issues2.html Figure 4.5 shows the result of this quer y. Not ice that the html/template package aut omat ically HTML-es cap ed the tit les so that the y appear literal ly. Had we used the text/template package by mistake , the four-ch arac ter str ing "&lt;" would have been rendered as a less-t han ch arac ter '<', and the str ing "<link>" would have becom e a link element, chang ing the st ruc ture of the HTML document and perhaps comp romising its sec urity.

We can sup press this aut o-es caping beh avior for fields that contain trusted HTML dat a by using the named str ing typ e template.HTML instead of string. Simi lar named typ es exist for tr usted JavaS crip t, CSS, and URLs. The program below demon strates the princip le by using two fields wit h the same value but dif ferent typ es: A is a str ing and B is a template.HTML.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 4.6. TEXT AND HTML TEMPLATES

117

Figure 4.5. HTML met acharac ters in issue tit les are cor rec tly displ aye d.

gopl.io/ch4/autoescape

func main() {

const templ = `<p>A: {{.A}}</p><p>B: {{.B}}</p>`

t := template.Must(template.New("escape").Parse(templ)) var data struct {

A string

// untrusted plain text

B template.HTML // trusted HTML

}

data.A = "<b>Hello!</b>"

data.B = "<b>Hello!</b>"

if err := t.Execute(os.Stdout, data); err != nil {

ptg16091132

log.Fatal(err)

}

}

Figure 4.6 shows the templ ate’s out put as it app ears in a brows er. We can see that A was su bjec t to escaping but B was not.

Figure 4.6. St ring values are HTML-es cap ed but template.HTML values are not.

We have space here to show only the most basic features of the templ ate system. As always, for more infor mat ion, cons ult the package documentation: $ go doc text/template

$ go doc html/template

Exercis e 4.14: Create a web ser ver that quer ies GitHub once and then allows nav igation of the list of bug rep orts, milestones, and users.

This page intentionally left blank ptg16091132

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

5

Functions

A function lets us wrap up a sequence of statements as a unit that can be cal le d from els e where in a program, perhaps multiple times. Functions make it possible to bre ak a big job int o smal ler pieces that mig ht wel l be writt en by dif ferent people sep arated by bot h time and space.

A function hides its implementation det ails fro m its users. For all of these reasons, functions ptg16091132

are a critical par t of any programming langu age.

We’ve seen many functions already. Now let’s take time for a more thoro ugh dis cussion. The running example of this chapt er is a web craw ler, that is, the component of a web searc h engine responsible for fetching web pages, discov ering the lin ks within them, fetching the pages identified by those lin ks, and so on. A we b craw ler gives us amp le opp ortunity to explore recursion, anonymou s func tions, error handling, and asp ects of functions that are unique to Go.

5.1. Function Declarations

A function declarat ion has a name, a list of parameters, an opt ion al list of results, and a body: func name(parameter-list) (result-list) {

body

}

The parameter list specifies the names and typ es of the function’s para meters, which are the lo cal variables whose values or ar gum ents are sup plie d by the cal ler. The result list specifies the typ es of the values that the function retur ns. If the function retur ns one unnamed result or no results at all, parenthes es are opt ion al and usu ally omitt ed. Leaving off the result list entire ly declares a function that does not retur n any value and is cal le d on ly for its effec ts. In the hypot func tion,

119

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

120

CHAPTER 5. FUNCTIONS

func hypot(x, y float64) float64 {

return math.Sqrt(x*x + y*y)

}

fmt.Println(hypot(3, 4)) // "5"

x and y are parameters in the declarat ion, 3 and 4 are arguments of the cal l, and the function returns a float64 value.

Like parameters, results may be named. In that cas e, each name declares a local variable initialize d to the zero value for its typ e.

A function that has a result list must end wit h a return st atement unless exe cut ion cle arly cannot reach the end of the function, perhaps because the function end s with a cal l to panic or an infinite for lo op wit h no break.

As we saw wit h hypot, a sequence of parameters or results of the same typ e can be fac tored so that the typ e itself is writt en only once. These two declarat ions are equivalent: func f(i, j, k int, s, t string)

{ /* ... */ }

func f(i int, j int, k int, s string, t string) { /* ... */ }

Here are four ways to declare a function wit h two parameters and one result, all of typ e int.

The blank identifier can be used to emp hasize that a parameter is unu sed.

func add(x int, y int) int

{ return x + y }

func sub(x, y int) (z int)

{ z = x - y; return }

ptg16091132

func first(x int, _ int) int { return x }

func zero(int, int) int

{ return 0 }

fmt.Printf("%T\n", add)

// "func(int, int) int"

fmt.Printf("%T\n", sub)

// "func(int, int) int"

fmt.Printf("%T\n", first) // "func(int, int) int"

fmt.Printf("%T\n", zero)

// "func(int, int) int"

The typ e of a function is som etimes cal le d its sig nat ure. Two functions have the same typ e or sig nature if the y have the same sequence of parameter typ es and the same sequence of result types. The names of parameters and results don’t affec t the typ e, nor does whether or not the y were declare d using the fac tored for m.

Ev ery function cal l mu st provide an argument for each parameter, in the order in which the parameters were declare d. Go has no con cept of defau lt parameter values, nor any way to sp ecif y arguments by name, so the names of parameters and results don’t matter to the cal ler except as documentation.

Parameters are local variables wit hin the body of the function, wit h their initial values set to the arguments sup plie d by the cal ler. Function parameters and named results are var iables in the same lexic al block as the function’s out ermost local variables.

Arguments are passed by value, so the function receives a copy of each argument; modification s to the copy do not affec t the cal ler. How ever, if the argument contains som e kind of reference, like a point er, slice, map, function, or channel, then the cal ler may be affec ted by any mo dification s the function makes to var iables in direc tly referred to by the argument.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.2. RECURSION

121

Yo u may occ asionally encount er a function declarat ion wit hout a body, indic ating that the func tion is imp lemente d in a langu age other than Go. Such a declarat ion defines the function sig nature.

package math

func Sin(x float64) float64 // implemented in assembly language 5.2. Recursion

Func tions may be re cur siv e, that is, the y may cal l themselves, either direc tly or indirec tly.

Re cursion is a pow erful technique for many pro blems, and of cours e it’s ess ent ial for processing rec ursive dat a st ruc tures. In Sec tion 4.4, we used rec ursion over a tree to imp lement a simple ins ertion sor t. In this sec tion, we’ll use it again for processing HTML documents.

The example program below uses a non-stand ard package, golang.org/x/net/html, which prov ides an HTML parser. The golang.org/x/... repositories hold packages designe d and maintained by the Go team for app lic ations such as net wor king, int ernat ionalize d text processing, mobile platfor ms, image manipu lat ion, cryptography, and develop er tools. These packages are not in the stand ard librar y becaus e they’re still under deve lopment or because they’re rarely needed by the maj ority of Go programmers.

ptg16091132

The par ts of the golang.org/x/net/html API that we’ll need are shown below. The function html.Parse re ads a sequence of bytes, parses them, and retur ns the root of the HTML document tre e, which is an html.Node. HTML has several kinds of nodes—text, comments, and so on—but here we are con cer ned only wit h el ement no des of the for m <name key='value'>.

golang.org/x/net/html

package html

type Node struct {

Type

NodeType

Data

string

Attr

[]Attribute

FirstChild, NextSibling *Node

}

type NodeType int32

const (

ErrorNode NodeType = iota

TextNode

DocumentNode

ElementNode

CommentNode

DoctypeNode

)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

122

CHAPTER 5. FUNCTIONS

type Attribute struct {

Key, Val string

}

func Parse(r io.Reader) (*Node, error)

The main func tion parses the stand ard inp ut as HTML, extrac ts the lin ks using a rec ursive visit func tion, and prints each dis cov ere d lin k:

gopl.io/ch5/findlinks1

// Findlinks1 prints the links in an HTML document read from standard input.

package main

import (

"fmt"

"os"

"golang.org/x/net/html"

)

func main() {

doc, err := html.Parse(os.Stdin)

if err != nil {

fmt.Fprintf(os.Stderr, "findlinks1: %v\n", err) os.Exit(1)

}

ptg16091132

for _, link := range visit(nil, doc) {

fmt.Println(link)

}

}

The visit func tion traverses an HTML node tre e, ext racts the lin k from the href attr ibute of each an chor element <a href='...'>, app end s the lin ks to a slice of str ings, and retur ns the resu lting slice:

// visit appends to links each link found in n and returns the result.

func visit(links []string, n *html.Node) []string {

if n.Type == html.ElementNode && n.Data == "a" {

for _, a := range n.Attr {

if a.Key == "href" {

links = append(links, a.Val)

}

}

}

for c := n.FirstChild; c != nil; c = c.NextSibling {

links = visit(links, c)

}

return links

}

To des cend the tre e for a node n, visit re cursive ly cal ls its elf for each of n’s chi ldren, which are held in the FirstChild lin ked list.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.2. RECURSION

123

Let’s run findlinks on the Go hom e page , piping the out put of fetch (§1.5) to the inp ut of findlinks. We’ve edite d the out put slig htly for brevit y.

$ go build gopl.io/ch1/fetch

$ go build gopl.io/ch5/findlinks1

$ ./fetch https://golang.org | ./findlinks1

#

/doc/

/pkg/

/help/

/blog/

http://play.golang.org/

//tour.golang.org/

https://golang.org/dl/

//blog.golang.org/

/LICENSE

/doc/tos.html

http://www.google.com/intl/en/policies/privacy/

No tice the var iety of for ms of lin ks that appear in the page. Later we’ll see how to res olve them rel ative to the bas e URL, https://golang.org, to make abs olut e URLs.

The next program uses rec ursion over the HTML node tre e to print the str ucture of the tre e in outline. As it encount ers each element, it pushes the element’s tag ont o a stack, then prints the st ack.

ptg16091132

gopl.io/ch5/outline

func main() {

doc, err := html.Parse(os.Stdin)

if err != nil {

fmt.Fprintf(os.Stderr, "outline: %v\n", err)

os.Exit(1)

}

outline(nil, doc)

}

func outline(stack []string, n *html.Node) {

if n.Type == html.ElementNode {

stack = append(stack, n.Data) // push tag

fmt.Println(stack)

}

for c := n.FirstChild; c != nil; c = c.NextSibling {

outline(stack, c)

}

}

No te one subtlet y: alt hough outline ‘‘pu shes’’ an element on stack, there is no cor responding pop. When outline calls its elf rec ursively, the cal le e re ceives a copy of stack. Alt hough the callee may append elements to this slice, modif ying its underly ing array and perhaps even al locating a new array, it doesn’t modif y the initial elements that are visible to the cal ler, so when the function retur ns, the cal ler’s stack is as it was before the cal l.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

124

CHAPTER 5. FUNCTIONS

Here’s the out line of https://golang.org, again edite d for bre vit y: $ go build gopl.io/ch5/outline

$ ./fetch https://golang.org | ./outline

[html]

[html head]

[html head meta]

[html head title]

[html head link]

[html body]

[html body div]

[html body div]

[html body div div]

[html body div div form]

[html body div div form div]

[html body div div form div a]

...

As you can see by exp erimenting wit h outline, most HTML documents can be pro cessed with only a few level s of rec ursion, but it’s not hard to con str uct pat holog ical web pages that re quire ext remely deep rec ursion.

Many programming langu age imp lementation s us e a fixe d-size function cal l st ack; sizes fro m 64KB to 2MB are typic al. Fixed-size stacks imp ose a limit on the depth of rec ursion, so one ptg16091132

mu st be caref ul to avoid a sta ck overflow when traversing large dat a st ruc tures rec ursively; fixe d-size stacks may even pos e a sec urity risk. In contrast, typic al Go imp lementation s us e var iable-size stacks that start small and grow as needed up to a limit on the order of a gigabyte.

This lets us use rec ursion safely and wit hout wor rying about overflow.

Exercis e 5.1: Change the findlinks prog ram to traverse the n.FirstChild lin ked list using re cursive cal ls to visit instead of a loop.

Exercis e 5.2: Wr ite a function to popu late a mapping fro m element names—p, div, span, and so on—to the number of elements wit h that name in an HTML document tre e.

Exercis e 5.3: Wr ite a function to print the contents of all text nodes in an HTML document tree. Do not des cend int o <script> or <style> elements, since their cont ents are not visible in a web brows er.

Exercis e 5.4: Extend the visit func tion so that it ext racts other kinds of lin ks from the document, such as images, scr ipts, and sty le she ets.

5.3. Multiple Return Values

A function can retur n more than one result. We’ve seen many examples of functions fro m st and ard packages that retur n two values, the desired computation al result and an error value or boole an that indic ates whether the computation wor ked. The next example shows how to wr ite one of our own.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.3. MULTIPLE RETURN VALUES

125

The program below is a var iat ion of findlinks that makes the HTTP request its elf so that we no lon g er need to run fetch. Because the HTTP and parsing operat ions can fail, findLinks de clares two results: the list of dis cov ere d lin ks and an error. Incidentally, the HTML parser can usu ally recov er fro m bad inp ut and con str uct a document containing error nodes, so Parse rarely fai ls; when it does, it’s typic ally due to underly ing I/O erro rs.

gopl.io/ch5/findlinks2

func main() {

for _, url := range os.Args[1:] {

links, err := findLinks(url)

if err != nil {

fmt.Fprintf(os.Stderr, "findlinks2: %v\n", err) continue

}

for _, link := range links {

fmt.Println(link)

}

}

}

// findLinks performs an HTTP GET request for url, parses the

// response as HTML, and extracts and returns the links.

func findLinks(url string) ([]string, error) {

resp, err := http.Get(url)

ptg16091132

if err != nil {

return nil, err

}

if resp.StatusCode != http.StatusOK {

resp.Body.Close()

return nil, fmt.Errorf("getting %s: %s", url, resp.Status)

}

doc, err := html.Parse(resp.Body)

resp.Body.Close()

if err != nil {

return nil, fmt.Errorf("parsing %s as HTML: %v", url, err)

}

return visit(nil, doc), nil

}

There are four retur n st atements in findLinks, each of which retur ns a pair of values. The firs t thre e returns cause the function to pass the underly ing error s from the http and html packages on to the cal ler. In the firs t case, the error is retur ned unchange d; in the secon d and third, it is aug mente d with addition al context infor mat ion by fmt.Errorf (§7.8). If findLinks is successf ul, the final retur n st atement retur ns the slice of lin ks, with no error.

We must ens ure that resp.Body is clos ed so that net wor k resources are pro perly released even in cas e of error. Go’s garb age col lec tor rec ycles unus ed memor y, but do not assume it will re lease unu sed operat ing system res ources like open files and net wor k connec tion s. The y should be clos ed explicitly.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

126

CHAPTER 5. FUNCTIONS

The result of cal ling a multi-value d func tion is a tup le of values. The cal ler of such a function mu st explicitly assig n the values to var iables if any of them are to be used: links, err := findLinks(url)

To ignore one of the values, assig n it to the blank identifier : links, _ := findLinks(url) // errors ignored

The result of a multi-value d call may its elf be retur ned fro m a (mu lti-value d) calling function, as in this function that beh aves like findLinks but logs its argument: func findLinksLog(url string) ([]string, error) {

log.Printf("findLinks %s", url)

return findLinks(url)

}

A multi-value d call may appear as the sole argument when cal ling a function of multiple parameters. Alt hough rarely used in pro duc tion code, this feature is som etimes convenient during debug ging since it lets us print all the results of a cal l using a single statement. The two pr int statements below have the same effec t.

log.Println(findLinks(url))

ptg16091132

links, err := findLinks(url)

log.Println(links, err)

We ll-chosen names can document the sig nificance of a function’s results. Names are par tic ularly valuable when a function retur ns multiple results of the same typ e, like func Size(rect image.Rectangle) (width, height int)

func Split(path string) (dir, file string)

func HourMinSec(t time.Time) (hour, minute, second int) but it’s not always necessary to name multiple results solely for documentation. For ins tance, conv ent ion dic tates that a final bool resu lt indic ates success; an error resu lt often needs no explanation.

In a function wit h name d resu lts, the operands of a retur n st atement may be omitt ed. This is called a bare return.

// CountWordsAndImages does an HTTP GET request for the HTML

// document url and returns the number of words and images in it.

func CountWordsAndImages(url string) (words, images int, err error) {

resp, err := http.Get(url)

if err != nil {

return

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.4. ERRORS

127

doc, err := html.Parse(resp.Body)

resp.Body.Close()

if err != nil {

err = fmt.Errorf("parsing HTML: %s", err)

return

}

words, images = countWordsAndImages(doc)

return

}

func countWordsAndImages(n *html.Node) (words, images int) { /* ... */ }

A bare retur n is a shorthand way to retur n each of the named result var iables in order, so in the function above , each retur n st atement is equivalent to return words, images, err

In functions like this one, wit h many retur n st atements and several results, bare retur ns can re duce code dup lic ation, but the y rarely make code easier to underst and. For ins tance, it’s not obviou s at firs t glance that the two early retur ns are equivalent to return 0, 0, err (b ecaus e the result var iables words and images are initialize d to their zero values) and that the final return is equivalent to return words, images, nil. For this reason, bare retur ns are best us ed sparing ly.

Exercis e 5.5: Implement countWordsAndImages. (See Exercis e 4.9 for word-split ting.) ptg16091132

Exercis e 5.6: Mo dif y the corner func tion in gopl.io/ch3/surface (§3.2) to use named resu lts and a bare retur n st atement.

5.4. Errors

Some functions always succe e d at their task. For example, strings.Contains and strconv.FormatBool have well-define d resu lts for all possible argument values and cannot fai l—

barring cat astrop hic and unp redic table scenar ios li ke running out of memor y, where the sy mpt om is far fro m the cause and fro m which there’s lit tle hop e of recov ery.

Ot her functions always succe e d so lon g as their pre con dit ion s are met. For example, the time.Date func tion always cons tructs a time.Time from its components—ye ar, month, and so on—un less the last argument (the time zon e) is nil, in which cas e it panics. This panic is a sure sig n of a bug in the cal ling code and should never happen in a wel l-w rit ten program.

Fo r many other functions, even in a wel l-w rit ten program, success is not assure d becaus e it dep ends on fac tor s beyond the programmer’s control . Any function that does I/O, for example, must confront the possibi lit y of error, and only a naïve programmer belie ves a simple read or write cannot fai l. Indeed, it’s when the most reliable operat ions fai l unexp ectedly that we most need to know why.

Er ror s are thu s an important par t of a package’s API or an app lic ation’s user int erface, and fai lure is just one of several expec ted beh avior s. This is the appro ach Go takes to error handling.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

128

CHAPTER 5. FUNCTIONS

A function for which fai lure is an exp ected beh avior retur ns an addition al result, conv ent ional ly the last one. If the fai lure has only one possible cause, the result is a boole an, usually called ok, as in this example of a cache looku p that always succe e ds unless there was no ent ry for that key :

value, ok := cache.Lookup(key)

if !ok {

// ...cache[key] does not exist...

}

Mo re often, and esp eci ally for I/O, the fai lure may have a var iety of causes for which the cal ler wi l l ne e d an explanation. In such cas es, the typ e of the addition al result is error.

The bui lt-in typ e error is an int erface typ e. We’ll see more of what this means and its implication s for error handling in Chapt er 7. For now it’s enoug h to know that an error may be nil or non-ni l, that nil imp lies success and non-ni l implies failure , and that a non-ni l error has an erro r mess age str ing which we can obt ain by cal ling its Error method or print by cal ling fmt.Println(err) or fmt.Printf("%v", err).

Us ual ly when a function retur ns a non-ni l er ror, its other results are undefine d and should be ig nored. How ever, a few functions may retur n partial results in erro r cases. For example, if an er ror occ urs whi le re ading fro m a file, a cal l to Read returns the number of bytes it was able to re ad an d an error value descr ibing the pro blem. For cor rec t behavior, som e callers may need ptg16091132

to pro cess the incomplete dat a before handling the error, so it is imp ortant that such functions clearly document their results.

Go’s appro ach sets it apart from many other langu ages in which fai lures are rep orted using ex cepti ons, not ordinar y values. Alt hough Go does have an exception mechanism of sor ts, as we will see in Sec tion 5.9, it is used only for rep orting truly unexp ected error s that indic ate a bug, not the routine error s that a robust program should be bui lt to exp ect.

The reason for this desig n is that exception s tend to ent ang le the des crip tion of an error wit h the control flow required to handle it, often leading to an undesirable out com e: routine error s are rep orted to the end user in the for m of an incomprehensible stack trace, full of infor mat ion about the str ucture of the program but lacking int ellig ible context about what went wro ng.

By contrast, Go programs use ordinar y cont rol-flow mechanisms like if and return to resp ond to error s. This sty le undeni ably demands that more att ent ion be paid to error-handling log ic, but that is pre cis ely the point.

5.4.1. Error-Handling Strategies

Wh en a function cal l returns an error, it’s the cal ler’s responsibilit y to check it and take appropriate action. Dep ending on the sit uat ion, there may be a numb er of possibi lit ies. Let’s take a look at five of them.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.4. ERRORS

129

First, and most common, is to prop agate the error, so that a fai lure in a subro utine becom es a fai lure of the cal ling routine. We saw examples of this in the findLinks func tion of Section 5.3. If the cal l to http.Get fai ls, findLinks returns the HTTP error to the cal ler without fur ther ado:

resp, err := http.Get(url)

if err != nil {

return nil, err

}

In contrast, if the cal l to html.Parse fai ls, findLinks do es not retur n the HTML parser’s er ror direc tly because it lacks two cruci al pieces of infor mat ion: that the error occ urre d in the pars er, and the URL of the document that was being parsed. In this cas e, findLinks const ruc ts a new error message that includes bot h pieces of infor mat ion as wel l as the underly ing pars e er ror :

doc, err := html.Parse(resp.Body)

resp.Body.Close()

if err != nil {

return nil, fmt.Errorf("parsing %s as HTML: %v", url, err)

}

The fmt.Errorf func tion for mats an erro r mess age using fmt.Sprintf and retur ns a new error

ptg16091132

value. We use it to bui ld des crip tive error s by successive ly prefixing addition al context infor mat ion to the original error message . Wh en the error is ult imate ly handled by the prog ram’s main func tion, it should provide a cle ar caus al chain fro m the root pro blem to the ov eral l fai lure, reminiscent of a NASA accident inv est igat ion: genesis: crashed: no parachute: G-switch failed: bad relay orientation Becaus e er ror messages are fre quently chained toget her, message str ings should not be capit alize d and newlines should be avoide d. The resulting error s may be lon g, but the y wi l l be self-cont ained when found by tools like grep.

Wh en desig ning error messages, be delib erate, so that each one is a meaningf ul des crip tion of the pro blem wit h sufficient and relevant det ail, and be con sistent, so that error s returned by the same function or by a gro up of functions in the same package are simi lar in for m and can be dealt wit h in the same way.

Fo r example, the os package guarante es that every error retur ned by a file operat ion, such as os.Open or the Read, Write, or Close methods of an open file, des crib es not just the nature of the fai lure (permission denie d, no such direc tor y, and so on) but als o the name of the file, so the cal ler ne e dn’t include this infor mat ion in the error message it con str ucts.

In general, the cal l f(x) is responsible for rep orting the att emp ted operat ion f and the argument value x as the y re late to the context of the error. The cal ler is responsible for adding fur-ther infor mat ion that it has but the cal l f(x) do es not, such as the URL in the cal l to html.Parse ab ove .

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

130

CHAPTER 5. FUNCTIONS

Let’s mov e on to the secon d st rateg y for handling error s. For error s that represent transient or unpredic table pro blems, it may make sense to re try the fai le d op erat ion, possibly wit h a del ay between tries, and perhaps wit h a limit on the number of att emp ts or the time spent trying before giv ing up ent ire ly.

gopl.io/ch5/wait

// WaitForServer attempts to contact the server of a URL.

// It tries for one minute using exponential back-off.

// It reports an error if all attempts fail.

func WaitForServer(url string) error {

const timeout = 1 * time.Minute

deadline := time.Now().Add(timeout)

for tries := 0; time.Now().Before(deadline); tries++ {

_, err := http.Head(url)

if err == nil {

return nil // success

}

log.Printf("server not responding (%s); retrying...", err) time.Sleep(time.Second << uint(tries)) // exponential back-off

}

return fmt.Errorf("server %s failed to respond after %s", url, timeout)

}

ptg16091132

Third, if progress is imp ossible, the cal ler can print the error and stop the program gracef ully, but this cours e of action should general ly be res erve d for the main package of a program.

Librar y func tions should usu ally pro pagate error s to the cal ler, unless the error is a sig n of an internal inconsistenc y—that is, a bug .

// (In function main.)

if err := WaitForServer(url); err != nil {

fmt.Fprintf(os.Stderr, "Site is down: %v\n", err) os.Exit(1)

}

A more convenient way to achie ve the same effec t is to cal l log.Fatalf. As wit h al l the log func tions, by defau lt it prefixes the time and date to the error message .

if err := WaitForServer(url); err != nil {

log.Fatalf("Site is down: %v\n", err)

}

The defau lt for mat is helpf ul in a lon g-r unning ser ver, but less so for an int erac tive tool: 2006/01/02 15:04:05 Site is down: no such domain: bad.gopl.io Fo r a more att rac tive out put, we can set the prefix used by the log package to the name of the command, and sup press the displ ay of the date and time: log.SetPrefix("wait: ")

log.SetFlags(0)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.4. ERRORS

131

Fo urth, in som e cases, it’s sufficient just to log the error and then continue, perhaps wit h re duce d func tionality. Again there’s a choice bet ween using the log package, which adds the usual prefix:

if err := Ping(); err != nil {

log.Printf("ping failed: %v; networking disabled", err)

}

and print ing direc tly to the stand ard error stream:

if err := Ping(); err != nil {

fmt.Fprintf(os.Stderr, "ping failed: %v; networking disabled\n", err)

}

(A ll log func tions app end a newline if one is not already present.) And fifth and finally, in rare cas es we can safely ignore an error ent ire ly: dir, err := ioutil.TempDir("", "scratch") if err != nil {

return fmt.Errorf("failed to create temp dir: %v", err)

}

// ...use temp dir...

os.RemoveAll(dir) // ignore errors; $TMPDIR is cleaned periodically ptg16091132

The cal l to os.RemoveAll may fai l, but the program ignores it because the operat ing system periodic ally cle ans out the temporar y direc tor y. In this cas e, dis carding the error was intention al, but the program logic wou ld be the same had we for g ott en to deal wit h it. Get into the habit of con sider ing error s af ter every function cal l, and when you delib erately ignore one, do cument your int ent ion cle arly.

Er ror handling in Go has a par tic ular rhythm. After che cking an error, fai lure is usu ally dealt with before success. If fai lure causes the function to retur n, the log ic for success is not indente d within an else block but fol lows at the out er level. Functions tend to exhibit a common str ucture, wit h a ser ies of initial che cks to rej e ct error s, fol low ed by the subst ance of the func tion at the end, minimal ly indente d.

5.4.2. End of File (EOF)

Us ual ly, the var iety of error s that a function may retur n is int erest ing to the end user but not to the int ervening program logic. On occasion, how ever, a program must take dif ferent ac tions dep ending on the kind of error that has occ urre d. Con sider an attemp t to read n bytes of dat a from a file. If n is chosen to be the lengt h of the file, any error represents a fai lure. On the other hand, if the cal ler repeatedly tries to read fixe d-size chunks unt i l the file is exhausted, the cal ler mu st respond dif ferently to an end-of-file condit ion than it does to all other erro rs.

Fo r this reason, the io package guarante es that any read fai lure caused by an end-of-file condition is always rep orted by a distinguishe d er ror, io.EOF, which is define d as fol lows:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

132

CHAPTER 5. FUNCTIONS

package io

import "errors"

// EOF is the error returned by Read when no more input is available.

var EOF = errors.New("EOF")

The cal ler can detec t this con dit ion using a simple comparison, as in the loop below, which re ads runes fro m the stand ard inp ut. (The charcount prog ram in Sec tion 4.3 provides a more comp lete example.)

in := bufio.NewReader(os.Stdin)

for {

r, _, err := in.ReadRune()

if err == io.EOF {

break // finished reading

}

if err != nil {

return fmt.Errorf("read failed: %v", err)

}

// ...use r...

}

Since in an end-of-file condit ion there is no infor mat ion to rep ort besides the fac t of it, io.EOF

has a fixe d er ror message , "EOF". For other erro rs, we may need to rep ort bot h the quality and ptg16091132

qu ant ity of the error, so to speak, so a fixe d er ror value will not do. In Sec tion 7.11, we’ll pres ent a more systematic way to distinguish cer tain erro r values fro m ot hers.

5.5. Function Values

Func tions are first-class values in Go: like other values, function values have typ es, and the y may be assig ned to var iables or passed to or retur ned fro m func tions. A func tion value may be cal le d li ke any other function. For example:

func square(n int) int

{ return n * n }

func negative(n int) int

{ return -n }

func product(m, n int) int { return m * n }

f := square

fmt.Println(f(3)) // "9"

f = negative

fmt.Println(f(3))

// "-3"

fmt.Printf("%T\n", f) // "func(int) int"

f = product // compile error: can't assign f(int, int) int to f(int) int The zero value of a function typ e is nil. Cal ling a nil function value causes a panic: var f func(int) int

f(3) // panic: call of nil function

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.5. FUNCTION VALUES

133

Func tion values may be compare d with nil:

var f func(int) int

if f != nil {

f(3)

}

but the y are not comparable, so the y may not be compare d against each other or used as keys in a map.

Func tion values let us parameter ize our functions over not just dat a, but beh avior too. The st and ard librar ies cont ain many examples. For ins tance, strings.Map applies a function to each charac ter of a str ing , joining the results to make another str ing .

func add1(r rune) rune { return r + 1 }

fmt.Println(strings.Map(add1, "HAL-9000")) // "IBM.:111"

fmt.Println(strings.Map(add1, "VMS"))

// "WNT"

fmt.Println(strings.Map(add1, "Admix"))

// "Benjy"

The findLinks func tion fro m Section 5.2 uses a helper function, visit, to visit all the nodes in an HTML document and app l y an action to each one. Using a function value, we can sep arate the log ic for tre e traversal fro m the log ic for the action to be app lie d to each node, letting us reuse the traversal wit h dif ferent actions.

ptg16091132

gopl.io/ch5/outline2

// forEachNode calls the functions pre(x) and post(x) for each node

// x in the tree rooted at n. Both functions are optional.

// pre is called before the children are visited (preorder) and

// post is called after (postorder).

func forEachNode(n *html.Node, pre, post func(n *html.Node)) {

if pre != nil {

pre(n)

}

for c := n.FirstChild; c != nil; c = c.NextSibling {

forEachNode(c, pre, post)

}

if post != nil {

post(n)

}

}

The forEachNode func tion accepts two function arguments, one to cal l before a node’s chi ldren are visit ed and one to cal l af ter. This arrangement gives the cal ler a gre at deal of flexibi lit y. For example, the functions startElement and endElement pr int the start and end tags of an HTML element like <b>...</b>:

var depth int

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

134

CHAPTER 5. FUNCTIONS

func startElement(n *html.Node) {

if n.Type == html.ElementNode {

fmt.Printf("%*s<%s>\n", depth*2, "", n.Data) depth++

}

}

func endElement(n *html.Node) {

if n.Type == html.ElementNode {

depth--

fmt.Printf("%*s</%s>\n", depth*2, "", n.Data)

}

}

The functions als o indent the out put using another fmt.Printf tr ick. The * adverb in %*s pr ints a str ing padde d with a var iable number of spaces. The width and the str ing are prov ide d by the arguments depth*2 and "".

If we cal l forEachNode on an HTML document, like this: forEachNode(doc, startElement, endElement)

we get a more elaborate var iat ion on the out put of our earlier outline prog ram: $ go build gopl.io/ch5/outline2

$ ./outline2 http://gopl.io

ptg16091132

<html>

<head>

<meta>

</meta>

<title>

</title>

<style>

</style>

</head>

<body>

<table>

<tbody>

<tr>

<td>

<a>

<img>

</img>

...

Exercis e 5.7: De velop startElement and endElement into a general HTML prett y-print er.

Pr int comment nodes, text nodes, and the att ribut es of each element (<a href='...'>). Use short for ms like <img/> instead of <img></img> when an element has no chi ldren. Write a test to ens ure that the out put can be parsed successf ully. (See Chapt er 11.) Exercis e 5.8: Mo dif y forEachNode so that the pre and post func tions retur n a boole an resu lt indic ating whether to continue the traversal. Use it to write a function ElementByID with the

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.6. ANONYMOUS FUNCTIONS

135

fo llowing sig nature that find s the firs t HTML element wit h the specified id attr ibute. The func tion should stop the traversal as soon as a match is found.

func ElementByID(doc *html.Node, id string) *html.Node

Exercis e 5.9: Wr ite a function expand(s string, f func(string) string) string that repl aces each subst ring ‘‘$foo’’ within s by the text retur ned by f("foo").

5.6. Anonymous Functions

Name d func tions can be declare d on ly at the package level, but we can use a func tion lit era l to denot e a function value wit hin any expression. A func tion lit eral is writt en like a function de clarat ion, but wit hout a name fol low ing the func ke yword . It is an expression, and its value is cal le d an an ony mou s func tion.

Func tion lit erals let us define a function at its point of use. As an example, the earlier cal l to strings.Map can be rewritt en as

strings.Map(func(r rune) rune { return r + 1 }, "HAL-9000") Mo re imp ortantly, functions define d in this way have access to the ent ire lexic al enviro nment, so the inner function can refer to var iables fro m the enclosing function, as this example shows:

ptg16091132

gopl.io/ch5/squares

// squares returns a function that returns

// the next square number each time it is called.

func squares() func() int {

var x int

return func() int {

x++

return x * x

}

}

func main() {

f := squares()

fmt.Println(f()) // "1"

fmt.Println(f()) // "4"

fmt.Println(f()) // "9"

fmt.Println(f()) // "16"

}

The function squares returns another function, of typ e func() int. A call to squares creates a local var iable x and retur ns an anony mou s func tion that, each time it is cal le d, increments x and retur ns its squ are . A secon d call to squares would cre ate a secon d var iable x and return a new anony mou s func tion which increments that var iable.

The squares example demon strates that function values are not just code but can have state.

The anony mou s inner function can access and update the local variables of the enclosing

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

136

CHAPTER 5. FUNCTIONS

func tion squares. These hidden variable references are why we classif y func tions as reference types and why function values are not comparable. Function values like these are imp lement ed using a technique cal le d cl osure s, and Go programmers often use this ter m for function values.

Here again we see an example where the lifet ime of a var iable is not deter mined by its scope: the var iable x exists after squares has retur ned wit hin main, even thoug h x is hidden inside f.

As a som ewhat academic example of anony mou s func tions, con sider the pro blem of computing a sequence of computer science courses that sat isfies the prere quisite requirements of each one. The prere quisites are given in the prereqs table below, which is a mapping fro m each cours e to the list of cours es that must be completed before it.

gopl.io/ch5/toposort

// prereqs maps computer science courses to their prerequisites.

var prereqs = map[string][]string{

"algorithms": {"data structures"},

"calculus":

{"linear algebra"},

"compilers": {

"data structures",

"formal languages",

"computer organization",

},

ptg16091132

"data structures":

{"discrete math"},

"databases":

{"data structures"},

"discrete math":

{"intro to programming"},

"formal languages":

{"discrete math"},

"networks":

{"operating systems"},

"operating systems":

{"data structures", "computer organization"},

"programming languages": {"data structures", "computer organization"},

}

This kind of pro blem is known as top olog ical sor ting. Con ceptu ally, the prere quisite infor mat ion for ms a direc ted graph wit h a node for each cours e and edges fro m each cours e to the cours es that it dep ends on. The graph is acyclic: there is no pat h from a cours e that leads back to its elf. We can comp ute a valid sequence using depth-firs t search through the graph with the code below :

func main() {

for i, course := range topoSort(prereqs) {

fmt.Printf("%d:\t%s\n", i+1, course)

}

}

func topoSort(m map[string][]string) []string {

var order []string

seen := make(map[string]bool)

var visitAll func(items []string)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.6. ANONYMOUS FUNCTIONS

137

visitAll = func(items []string) {

for _, item := range items {

if !seen[item] {

seen[item] = true

visitAll(m[item])

order = append(order, item)

}

}

}

var keys []string

for key := range m {

keys = append(keys, key)

}

sort.Strings(keys)

visitAll(keys)

return order

}

Wh en an anony mou s func tion requires rec ursion, as in this example, we must firs t de clare a var iable, and then assig n the anony mou s func tion to that var iable. Had these two steps been combined in the declarat ion, the function lit eral wou ld not be wit hin the scope of the var iable visitAll so it wou ld have no way to cal l itself rec ursively: visitAll := func(items []string) {

ptg16091132

// ...

visitAll(m[item]) // compile error: undefined: visitAll

// ...

}

The out put of the toposort prog ram is shown below. It is deter minist ic, an often-desirable prop erty that doesn’t always come for fre e. Here, the values of the prereqs map are slices, not more maps, so their iterat ion order is deter minist ic, and we sor ted the keys of prereqs before making the initial cal ls to visitAll.

1:

intro to programming

2:

discrete math

3:

data structures

4:

algorithms

5:

linear algebra

6:

calculus

7:

formal languages

8:

computer organization

9:

compilers

10:

databases

11:

operating systems

12:

networks

13:

programming languages

Let’s retur n to our findLinks example. We’ve mov ed the lin k-ext raction function links.Extract to its own package, since we’ll use it again in Chapt er 8. We replace d the

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

138

CHAPTER 5. FUNCTIONS

visit func tion wit h an anonymou s func tion that app end s to the links slice direc tly, and used forEachNode to handle the traversal. Since Extract ne e ds only the pre func tion, it passes nil for the post argument.

gopl.io/ch5/links

// Package links provides a link-extraction function.

package links

import (

"fmt"

"net/http"

"golang.org/x/net/html"

)

// Extract makes an HTTP GET request to the specified URL, parses

// the response as HTML, and returns the links in the HTML document.

func Extract(url string) ([]string, error) {

resp, err := http.Get(url)

if err != nil {

return nil, err

}

if resp.StatusCode != http.StatusOK {

resp.Body.Close()

return nil, fmt.Errorf("getting %s: %s", url, resp.Status) ptg16091132

}

doc, err := html.Parse(resp.Body)

resp.Body.Close()

if err != nil {

return nil, fmt.Errorf("parsing %s as HTML: %v", url, err)

}

var links []string

visitNode := func(n *html.Node) {

if n.Type == html.ElementNode && n.Data == "a" {

for _, a := range n.Attr {

if a.Key != "href" {

continue

}

link, err := resp.Request.URL.Parse(a.Val)

if err != nil {

continue // ignore bad URLs

}

links = append(links, link.String())

}

}

}

forEachNode(doc, visitNode, nil)

return links, nil

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.6. ANONYMOUS FUNCTIONS

139

In ste ad of app ending the raw href attr ibute value to the links slice, this version parses it as a URL rel ative to the bas e URL of the document, resp.Request.URL. The resulting link is in abs olut e form, suitable for use in a cal l to http.Get.

Craw ling the web is, at its heart, a problem of graph traversal. The topoSort example showe d a depth-firs t traversal; for our web craw ler, we’ll use bre adt h-firs t traversal, at least initial ly. In Chapter 8, we’ll explore con cur rent traversal.

The function below enc apsulates the ess ence of a bre adt h-firs t traversal. The cal ler prov ides an initial list worklist of items to visit and a function value f to cal l for each item. Each item is identified by a str ing . The function f returns a list of new items to app end to the wor klist.

The breadthFirst func tion retur ns when all items have been visit ed. It maintains a set of st rings to ens ure that no item is visit ed twice.

gopl.io/ch5/findlinks3

// breadthFirst calls f for each item in the worklist.

// Any items returned by f are added to the worklist.

// f is called at most once for each item.

func breadthFirst(f func(item string) []string, worklist []string) {

seen := make(map[string]bool)

for len(worklist) > 0 {

items := worklist

worklist = nil

ptg16091132

for _, item := range items {

if !seen[item] {

seen[item] = true

worklist = append(worklist, f(item)...)

}

}

}

}

As we explained in passing in Chapt er 3, the argument ‘‘f(item)...’’ caus es all the items in the list retur ned by f to be app ended to the wor klist.

In our crawler, items are URLs. The crawl func tion we’ll sup ply to breadthFirst pr ints the URL, extrac ts its lin ks, and retur ns them so that the y to o are visit ed.

func crawl(url string) []string {

fmt.Println(url)

list, err := links.Extract(url)

if err != nil {

log.Print(err)

}

return list

}

To start the craw ler of f, we’ll use the command-line arguments as the initial URLs.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

140

CHAPTER 5. FUNCTIONS

func main() {

// Crawl the web breadth-first,

// starting from the command-line arguments.

breadthFirst(crawl, os.Args[1:])

}

Let’s craw l the web starting fro m https://golang.org. Here are som e of the resulting lin ks: $ go build gopl.io/ch5/findlinks3

$ ./findlinks3 https://golang.org

https://golang.org/

https://golang.org/doc/

https://golang.org/pkg/

https://golang.org/project/

https://code.google.com/p/go-tour/

https://golang.org/doc/code.html

https://www.youtube.com/watch?v=XCsL89YtqCs

http://research.swtch.com/gotour

https://vimeo.com/53221560

...

The pro cess end s when all reach able web pages have been craw led or the memory of the computer is exhausted.

Exercis e 5.10: Re writ e topoSort to use maps instead of slices and eliminate the initial sor t.

ptg16091132

Verify that the results, thoug h nondeter minist ic, are valid top olog ical order ings.

Exercis e 5.11: The ins tructor of the linear algebra course decides that calc ulu s is now a prerequisite. Extend the topoSort func tion to rep ort cyc les.

Exercis e 5.12: The startElement and endElement func tions in gopl.io/ch5/outline2

(§5.5) share a global var iable, depth. Tur n them into anony mou s func tions that share a var iable local to the outline func tion.

Exercis e 5.13: Mo dif y crawl to make local copies of the pages it find s, cre ating direc tor ies as ne cessary. Don’t make copies of pages that com e from a dif ferent dom ain. For example, if the or iginal page com es fro m golang.org, save all files fro m there, but exc lude ones fro m vimeo.com.

Exercis e 5.14: Us e the breadthFirst func tion to explore a dif ferent str ucture. For example, you cou ld use the cours e dep endencies fro m the topoSort example (a direc ted graph), the file system hierarchy on your computer (a tre e), or a list of bus or subway routes dow nlo ade d from your city gov ernment’s web sit e (an undirec ted graph).

5.6.1. Caveat: Capturing Iteration Variables

In this sec tion, we’ll look at a pit fal l of Go’s lexic al scop e rules that can cause sur prising results.

We urge you to underst and the pro blem before pro ceeding, because the trap can ensnare even exp erience d prog rammers.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.6. ANONYMOUS FUNCTIONS

141

Consider a program that must cre ate a set of direc tor ies and later remov e them. We can use a slice of function values to hold the cle an-up operat ions. (Fo r brevit y, we have omitt ed all error hand ling in this example.)

var rmdirs []func()

for _, d := range tempDirs() {

dir := d

// NOTE: necessary!

os.MkdirAll(dir, 0755) // creates parent directories too rmdirs = append(rmdirs, func() {

os.RemoveAll(dir)

})

}

// ...do some work...

for _, rmdir := range rmdirs {

rmdir() // clean up

}

Yo u may be won der ing why we assig ned the loop var iable d to a new local variable dir within the loop body, ins tead of just naming the loop var iable dir as in this subtly incor rec t var iant: var rmdirs []func()

for _, dir := range tempDirs() {

os.MkdirAll(dir, 0755)

ptg16091132

rmdirs = append(rmdirs, func() {

os.RemoveAll(dir) // NOTE: incorrect!

})

}

The reason is a con sequence of the scope rules for loop var iables. In the program immediate ly ab ove , the for lo op int roduces a new lexic al block in which the var iable dir is declare d. All func tion values create d by this loop ‘‘capt ure’’ and share the same var iable—an addressable storage locat ion, not its value at that par tic ular mom ent. The value of dir is updated in successive iterat ions, so by the time the cle anup functions are cal le d, the dir var iable has been up dated several times by the now-comp leted for lo op. Thu s dir holds the value fro m the final iterat ion, and con sequently all cal ls to os.RemoveAll wi l l attemp t to remov e the same direc tor y.

Fr equently, the inner variable int roduce d to wor k around this pro blem—dir in our example—

is given the exac t same name as the out er var iable of which it is a copy, leading to odd-lo oking but cruci al var iable decl arat ions like this:

for _, dir := range tempDirs() {

dir := dir // declares inner dir, initialized to outer dir

// ...

}

The risk is not unique to range-b ased for lo ops. The loop in the example below suf fers fro m the same pro blem due to unintended capture of the index variable i.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

142

CHAPTER 5. FUNCTIONS

var rmdirs []func()

dirs := tempDirs()

for i := 0; i < len(dirs); i++ {

os.MkdirAll(dirs[i], 0755) // OK

rmdirs = append(rmdirs, func() {

os.RemoveAll(dirs[i]) // NOTE: incorrect!

})

}

The pro blem of iterat ion var iable capture is most often encountere d when using the go st atement (Chapter 8) or wit h defer (w hich we will see in a mom ent) since bot h may del ay the exec ution of a function value until after the loop has finishe d. But the pro blem is not inherent to go or defer.

5.7. Variadic Functions

A variadic fun cti on is one that can be cal le d with var ying numbers of arguments. The most fami liar examples are fmt.Printf and its variants. Printf re quires one fixe d argument at the beginning, then accepts any number of subsequent arguments.

To declare a var iadic function, the typ e of the final parameter is pre ceded by an ellipsis, ‘‘...’’, ptg16091132

which indic ates that the function may be cal le d with any number of arguments of this typ e.

gopl.io/ch5/sum

func sum(vals ...int) int {

total := 0

for _, val := range vals {

total += val

}

return total

}

The sum func tion above retur ns the sum of zero or more int arguments. Wit hin the body of the function, the typ e of vals is an []int slice. When sum is cal le d, any number of values may be provide d for its vals parameter.

fmt.Println(sum())

//

"0"

fmt.Println(sum(3))

//

"3"

fmt.Println(sum(1, 2, 3, 4)) //

"10"

Implicitly, the cal ler al locates an array, copies the arguments into it, and passes a slice of the entire array to the function. The last cal l ab ove thu s behaves the same as the cal l below, which shows how to inv oke a var iadic function when the arguments are already in a slice: place an el lipsis after the final argument.

values := []int{1, 2, 3, 4}

fmt.Println(sum(values...)) // "10"

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.8. DEFERRED FUNCTION CALLS

143

Although the ...int parameter beh aves like a slice wit hin the function body, the typ e of a var iadic function is distinc t from the typ e of a function wit h an ordinar y slice parameter.

func f(...int) {}

func g([]int)

{}

fmt.Printf("%T\n", f) // "func(...int)"

fmt.Printf("%T\n", g) // "func([]int)"

Variadic functions are often used for str ing for matting . The errorf func tion below const ruc ts a for matte d er ror message wit h a line number at the beg inning. The suffix f is a widely fo llowe d naming convent ion for var iadic functions that accept a Printf-sty le for mat str ing .

func errorf(linenum int, format string, args ...interface{}) {

fmt.Fprintf(os.Stderr, "Line %d: ", linenum)

fmt.Fprintf(os.Stderr, format, args...)

fmt.Fprintln(os.Stderr)

}

linenum, name := 12, "count"

errorf(linenum, "undefined: %s", name) // "Line 12: undefined: count"

The interface{} type means that this function can accept any values at all for its final arguments, as we’ll explain in Chapt er 7.

Exercis e 5.15: Wr ite var iadic functions max and min, analogou s to sum. What should these ptg16091132

func tions do when cal le d with no arguments? Write var iants that require at least one argument.

Exercis e 5.16: Wr ite a var iadic version of strings.Join.

Exercis e 5.17: Wr ite a var iadic function ElementsByTagName that, given an HTML node tre e and zero or more names, retur ns all the elements that match one of those names. Here are two example cal ls:

func ElementsByTagName(doc *html.Node, name ...string) []*html.Node images := ElementsByTagName(doc, "img")

headings := ElementsByTagName(doc, "h1", "h2", "h3", "h4") 5.8. Deferred Function Calls

Our findLinks examples used the out put of http.Get as the inp ut to html.Parse. This works wel l if the content of the requeste d URL is indeed HTML, but many pages cont ain images, plain text, and other file for mats. Feeding such files into an HTML parser cou ld have undesirable effe cts.

The program below fetch es an HTML document and prints its tit le. The title func tion insp ects the Content-Type he ader of the ser ver’s respons e and retur ns an error if the document is not HTML.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

144

CHAPTER 5. FUNCTIONS

gopl.io/ch5/title1

func title(url string) error {

resp, err := http.Get(url)

if err != nil {

return err

}

// Check Content-Type is HTML (e.g., "text/html; charset=utf-8").

ct := resp.Header.Get("Content-Type")

if ct != "text/html" && !strings.HasPrefix(ct, "text/html;") {

resp.Body.Close()

return fmt.Errorf("%s has type %s, not text/html", url, ct)

}

doc, err := html.Parse(resp.Body)

resp.Body.Close()

if err != nil {

return fmt.Errorf("parsing %s as HTML: %v", url, err)

}

visitNode := func(n *html.Node) {

if n.Type == html.ElementNode && n.Data == "title" && n.FirstChild != nil {

fmt.Println(n.FirstChild.Data)

}

ptg16091132

}

forEachNode(doc, visitNode, nil)

return nil

}

Here’s a typic al session, slig htly edite d to fit:

$ go build gopl.io/ch5/title1

$ ./title1 http://gopl.io

The Go Programming Language

$ ./title1 https://golang.org/doc/effective_go.html

Effective Go - The Go Programming Language

$ ./title1 https://golang.org/doc/gopher/frontpage.png

title: https://golang.org/doc/gopher/frontpage.png

has type image/png, not text/html

Observe the dup lic ated resp.Body.Close() call, which ens ures that title clos es the network conne ction on all exe cut ion pat hs, including fai lures. As functions grow more complex and have to handle more error s, such dup lic ation of cle an-up log ic may becom e a maintenance pro blem. Let’s see how Go’s nov el defer me chanism makes things simpler.

Sy ntactic ally, a defer st atement is an ordinar y func tion or met hod cal l prefixe d by the ke yword defer. The function and argument expressions are evaluate d when the statement is exec ute d, but the actual cal l is deferred until the function that contains the defer st atement has finishe d, whether nor mal ly, by exe cut ing a retur n st atement or fal ling off the end, or abnormal ly, by panicking . Any number of cal ls may be defer red; the y are exe cut ed in the

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.8. DEFERRED FUNCTION CALLS

145

re verse of the order in which the y were defer red.

A defer st atement is often used wit h paired operat ions like open and clos e, conne ct and disconnec t, or lock and unlock to ens ure that res ources are released in all cas es, no matter how comp lex the control flow. The rig ht place for a defer st atement that releases a res ource is immediate ly after the res ource has been successf ully acquired. In the title func tion below, a single defer red cal l repl aces bot h previous cal ls to resp.Body.Close(): gopl.io/ch5/title2

func title(url string) error {

resp, err := http.Get(url)

if err != nil {

return err

}

defer resp.Body.Close()

ct := resp.Header.Get("Content-Type")

if ct != "text/html" && !strings.HasPrefix(ct, "text/html;") {

return fmt.Errorf("%s has type %s, not text/html", url, ct)

}

doc, err := html.Parse(resp.Body)

if err != nil {

return fmt.Errorf("parsing %s as HTML: %v", url, err) ptg16091132

}

// ...print doc's title element...

return nil

}

The same pattern can be used for other res ources beside net wor k connec tion s, for ins tance to clos e an open file:

io/ioutil

package ioutil

func ReadFile(filename string) ([]byte, error) {

f, err := os.Open(filename)

if err != nil {

return nil, err

}

defer f.Close()

return ReadAll(f)

}

or to unlock a mut ex (§9.2):

var mu sync.Mutex

var m = make(map[string]int)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

146

CHAPTER 5. FUNCTIONS

func lookup(key string) int {

mu.Lock()

defer mu.Unlock()

return m[key]

}

The defer st atement can also be used to pair ‘‘on ent ry’’ and ‘‘on exit’’ ac tions when debug ging a complex function. The bigSlowOperation func tion below cal ls trace immediate ly, which do es the ‘‘on ent ry’’ ac tion then retur ns a function value that, when cal le d, does the corresp onding ‘‘on exit’’ ac tion. By defer r ing a cal l to the retur ned function in this way, we can inst rument the ent ry point and all exit points of a function in a single statement and even pass values, like the start time, bet ween the two actions. But don’t for g et the final parenthes es in the defer st atement, or the ‘‘on ent ry’’ ac tion will happen on exit and the on-exit action won’t happ en at all!

gopl.io/ch5/trace

func bigSlowOperation() {

defer trace("bigSlowOperation")() // don't forget the extra parentheses

// ...lots of work...

time.Sleep(10 * time.Second) // simulate slow operation by sleeping

}

func trace(msg string) func() {

ptg16091132

start := time.Now()

log.Printf("enter %s", msg)

return func() { log.Printf("exit %s (%s)", msg, time.Since(start)) }

}

Each time bigSlowOperation is cal le d, it logs its entr y and exit and the elaps ed time bet ween them. (We used time.Sleep to simulate a slow operat ion.) $ go build gopl.io/ch5/trace

$ ./trace

2015/11/18 09:53:26 enter bigSlowOperation

2015/11/18 09:53:36 exit bigSlowOperation (10.000589217s) Deferred functions run af ter return statements have updated the function’s result var iables.

Becaus e an anonymou s func tion can access its enclosing function’s var iables, including named resu lts, a defer red anony mou s func tion can obs erve the function’s results.

Consider the function double:

func double(x int) int {

return x + x

}

By naming its result var iable and adding a defer st atement, we can make the function print its arguments and results each time it is cal le d.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.8. DEFERRED FUNCTION CALLS

147

func double(x int) (result int) {

defer func() { fmt.Printf("double(%d) = %d\n", x, result) }() return x + x

}

_ = double(4)

// Output:

// "double(4) = 8"

This trick is overk ill for a function as simple as double but may be useful in functions wit h many retur n st atements.

A defer red anony mou s func tion can even change the values that the enclosing function returns to its cal ler:

func triple(x int) (result int) {

defer func() { result += x }()

return double(x)

}

fmt.Println(triple(4)) // "12"

Becaus e defer red functions aren’t exe cut ed unt i l the ver y end of a function’s exe cut ion, a defer st atement in a loop des erves ext ra scrutiny. The code below cou ld run out of file des crip tor s since no file will be clos ed unt i l al l files have been pro cessed: ptg16091132

for _, filename := range filenames {

f, err := os.Open(filename)

if err != nil {

return err

}

defer f.Close() // NOTE: risky; could run out of file descriptors

// ...process f...

}

One solut ion is to mov e the loop body, including the defer st atement, int o anot her function that is cal le d on each iterat ion.

for _, filename := range filenames {

if err := doFile(filename); err != nil {

return err

}

}

func doFile(filename string) error {

f, err := os.Open(filename)

if err != nil {

return err

}

defer f.Close()

// ...process f...

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

148

CHAPTER 5. FUNCTIONS

The example below is an imp rov ed fetch prog ram (§1.5) that writes the HTTP respons e to a lo cal file instead of to the stand ard out put. It der ives the file name fro m the last component of the URL pat h, which it obt ains using the path.Base func tion.

gopl.io/ch5/fetch

// Fetch downloads the URL and returns the

// name and length of the local file.

func fetch(url string) (filename string, n int64, err error) {

resp, err := http.Get(url)

if err != nil {

return "", 0, err

}

defer resp.Body.Close()

local := path.Base(resp.Request.URL.Path)

if local == "/" {

local = "index.html"

}

f, err := os.Create(local)

if err != nil {

return "", 0, err

}

n, err = io.Copy(f, resp.Body)

// Close file, but prefer error from Copy, if any.

ptg16091132

if closeErr := f.Close(); err == nil {

err = closeErr

}

return local, n, err

}

The defer red cal l to resp.Body.Close should be fami liar by now. It’s tempt ing to use a second defer red cal l, to f.Close, to clos e the local file, but this wou ld be subtly wro ng because os.Create op ens a file for writing , creating it as needed. On many file systems, not ably NFS, wr ite error s are not rep orted immediate ly but may be postp one d until the file is clos ed. Fai lure to che ck the result of the clos e op erat ion cou ld cause seriou s data loss to go unnot ice d.

Ho wever, if bot h io.Copy and f.Close fai l, we should prefer to rep ort the error fro m io.Copy since it occ urre d firs t and is more likely to tel l us the root cause.

Exercis e 5.18: Wi thout chang ing its beh avior, rewrite the fetch func tion to use defer to clos e the writable file.

5.9. Panic

Go’s typ e system catch es many mistakes at compi le time, but others, like an out-of-b ounds ar ray access or nil point er dereference, require che cks at run time. When the Go runtime detec ts these mistakes, it pani cs.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.9. PANIC

149

During a typic al panic, nor mal exec ution stops, all defer red function cal ls in that goroutine are exec ute d, and the program crashes wit h a log message . This log message includes the pani c valu e, which is usu ally an error message of som e sort, and, for each goroutine, a sta ck tra c e showing the stack of function cal ls that were active at the time of the panic. This log message of ten has enough infor mat ion to diagnos e the root cause of the pro blem wit hout running the prog ram again, so it should always be include d in a bug rep ort about a panicking program.

No t al l panics come fro m the runtime. The bui lt-in panic func tion may be cal le d direc tly ; it accepts any value as an argument. A panic is often the best thing to do when som e ‘‘impossi-ble’’ situ ation happens, for ins tance, exe cut ion reach es a cas e that logic ally can’t happen: switch s := suit(drawCard()); s {

case "Spades":

// ...

case "Hearts":

// ...

case "Diamonds": // ...

case "Clubs":

// ...

default:

panic(fmt.Sprintf("invalid suit %q", s)) // Joker?

}

It’s good prac tice to ass ert that the pre con dit ion s of a function hold, but this can easi ly be don e to excess. Unless you can provide a more infor mat ive error message or detec t an erro r so oner, there is no point ass erting a con dit ion that the runtime will che ck for you .

ptg16091132

func Reset(x *Buffer) {

if x == nil {

panic("x is nil") // unnecessary!

}

x.elements = nil

}

Although Go’s panic mechanism res embles exception s in other langu ages, the sit uat ions in which panic is used are quite dif ferent. Since a panic causes the program to crash, it is general ly used for grave error s, such as a log ical inconsistenc y in the program; diligent programmers con sider any crash to be pro of of a bug in their code. In a robust program, ‘‘exp ected’’

er ror s, the kind that arise fro m incorrec t input, misconfigurat ion, or fai ling I/O, should be hand led gracef ully; the y are best dealt wit h using error values.

Consider the function regexp.Compile, which compi les a regu lar expression int o an efficient form for matching. It retur ns an error if cal le d with an ill-for med patt ern, but che cking this er ror is unnecessary and burdensome if the cal ler knows that a partic ular cal l cannot fai l. In such cas es, it’s reasonable for the cal ler to handle an error by panicking , since it is belie ved to be imp ossible.

Since most regu lar expressions are lit erals in the program source code, the regexp package prov ides a wrapper function regexp.MustCompile that does this che ck: package regexp

func Compile(expr string) (*Regexp, error) { /* ... */ }

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

150

CHAPTER 5. FUNCTIONS

func MustCompile(expr string) *Regexp {

re, err := Compile(expr)

if err != nil {

panic(err)

}

return re

}

The wrapper function makes it convenient for clients to initialize a package-le vel var iable wit h a compi led regu lar expression, like this:

var httpSchemeRE = regexp.MustCompile(`^https?:`) // "http:" or "https:"

Of course, MustCompile should not be cal le d with unt ruste d input values. The Must prefix is a common naming convent ion for functions of this kind, like template.Must in Sec tion 4.6.

Wh en a panic occ urs, all defer red functions are run in reverse order, starting wit h thos e of the topm ost function on the stack and pro ceeding up to main, as the program below demon strates:

gopl.io/ch5/defer1

func main() {

f(3)

}

ptg16091132

func f(x int) {

fmt.Printf("f(%d)\n", x+0/x) // panics if x == 0

defer fmt.Printf("defer %d\n", x)

f(x - 1)

}

Wh en run, the program prints the fol low ing to the stand ard out put: f(3)

f(2)

f(1)

defer 1

defer 2

defer 3

A panic occ urs dur ing the cal l to f(0), causing the three defer red cal ls to fmt.Printf to run.

Then the runtime ter minates the program, print ing the panic message and a stack dump to the stand ard error stream (simplified for clarity):

panic: runtime error: integer divide by zero

main.f(0)

src/gopl.io/ch5/defer1/defer.go:14

main.f(1)

src/gopl.io/ch5/defer1/defer.go:16

main.f(2)

src/gopl.io/ch5/defer1/defer.go:16

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.10. RECOVER

151

main.f(3)

src/gopl.io/ch5/defer1/defer.go:16

main.main()

src/gopl.io/ch5/defer1/defer.go:10

As we will see soon, it is possible for a function to recov er fro m a panic so that it does not terminate the program.

Fo r di agnost ic pur pos es, the runtime package lets the programmer dump the stack using the same machiner y. By defer r ing a cal l to printStack in main, gopl.io/ch5/defer2

func main() {

defer printStack()

f(3)

}

func printStack() {

var buf [4096]byte

n := runtime.Stack(buf[:], false)

os.Stdout.Write(buf[:n])

}

the fol low ing addition al text (again simplified for clarity) is print ed to the stand ard out put: goroutine 1 [running]:

ptg16091132

main.printStack()

src/gopl.io/ch5/defer2/defer.go:20

main.f(0)

src/gopl.io/ch5/defer2/defer.go:27

main.f(1)

src/gopl.io/ch5/defer2/defer.go:29

main.f(2)

src/gopl.io/ch5/defer2/defer.go:29

main.f(3)

src/gopl.io/ch5/defer2/defer.go:29

main.main()

src/gopl.io/ch5/defer2/defer.go:15

Re aders fami liar wit h exception s in other langu ages may be sur prised that runtime.Stack can print infor mat ion about functions that seem to have already been ‘‘unwound.’’ Go’s panic me chanism runs the defer red functions before it unw ind s the stack.

5.10. Recover

Gi ving up is usu ally the rig ht respons e to a panic, but not always. It mig ht be possible to re cov er in som e way, or at least cle an up the mess before quitt ing . Fo r example, a web ser ver that encounters an unexp ected pro blem could clos e the conne ction rat her than leave the client hang ing , and dur ing development, it mig ht rep ort the error to the client too.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

152

CHAPTER 5. FUNCTIONS

If the bui lt-in recover func tion is cal le d within a defer red function and the function containing the defer st atement is panicking , recover ends the cur rent state of panic and retur ns the panic value. The function that was panicking does not continue where it lef t of f but retur ns normal ly. If recover is cal le d at any other time, it has no effec t and retur ns nil.

To illustrate, con sider the development of a parser for a langu age. Even when it app ears to be work ing well, given the complexity of its job, bugs may still lurk in obs cure cor ner cas es. We mig ht prefer that, instead of crashing, the parser tur ns these panics into ordinar y pars e er ror s, perh aps wit h an extra message exhor ting the user to file a bug rep ort.

func Parse(input string) (s *Syntax, err error) {

defer func() {

if p := recover(); p != nil {

err = fmt.Errorf("internal error: %v", p)

}

}()

// ...parser...

}

The defer red function in Parse re cov ers fro m a panic, using the panic value to con str uct an er ror message; a fancier version mig ht include the ent ire cal l st ack using runtime.Stack. The defer red function then assig ns to the err resu lt, which is retur ned to the cal ler.

ptg16091132

Re cov ering indis criminately fro m panics is a dubious prac tice because the state of a package’s var iables after a panic is rarely wel l define d or documente d. Perh aps a crit ical update to a dat a st ruc ture was incomp lete, a file or net wor k connec tion was opene d but not clos ed, or a lock was acquired but not released. Fur thermore, by replacing a crash wit h, say, a line in a log file, indiscriminate recov ery may cause bugs to go unnot ice d.

Re cov ering fro m a panic wit hin the same package can help simplif y the handling of complex or unexp ected error s, but as a general rule, you should not att emp t to recov er fro m anot her package’s panic. Public APIs should rep ort fai lures as errors. Simi larly, you should not re cov er fro m a panic that may pass through a function you do not maintain, such as a cal ler-prov ide d callback, since you cannot reason about its safet y.

Fo r example, the net/http package provides a web ser ver that dispatch es incoming requests to user-prov ide d hand ler functions. Rat her than let a panic in one of these handlers kill the process, the ser ver cal ls recover, prints a stack trace, and continues ser ving. This is convenient in prac tice, but it does risk leaking res ources or leaving the fai le d hand ler in an unsp ecified state that cou ld lead to other pro blems.

Fo r al l the above reasons, it’s safest to recov er selec tive ly if at all. In other words, recov er only from panics that were int ended to be recov ere d from, which should be rare. This int ent ion can be encoded by using a dist inc t, unexp orted typ e for the panic value and testing whether the value retur ned by recover has that typ e. (We’ll see one way to do this in the next example.) If so, we rep ort the panic as an ordinar y error; if not, we cal l panic with the same value to resume the state of panic.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 5.10. RECOVER

153

The example below is a var iat ion on the title prog ram that rep orts an error if the HTML

do cument contains multiple <title> elements. If so, it aborts the rec ursion by cal ling panic with a value of the speci al type bailout.

gopl.io/ch5/title3

// soleTitle returns the text of the first non-empty title element

// in doc, and an error if there was not exactly one.

func soleTitle(doc *html.Node) (title string, err error) {

type bailout struct{}

defer func() {

switch p := recover(); p {

case nil:

// no panic

case bailout{}:

// "expected" panic

err = fmt.Errorf("multiple title elements")

default:

panic(p) // unexpected panic; carry on panicking

}

}()

// Bail out of recursion if we find more than one non-empty title.

forEachNode(doc, func(n *html.Node) {

if n.Type == html.ElementNode && n.Data == "title" && ptg16091132

n.FirstChild != nil {

if title != "" {

panic(bailout{}) // multiple title elements

}

title = n.FirstChild.Data

}

}, nil)

if title == "" {

return "", fmt.Errorf("no title element")

}

return title, nil

}

The defer red handler function cal ls recover, che cks the panic value, and rep orts an ordinar y er ror if the value was bailout{}. All other non-ni l values indic ate an unexp ected panic, in which cas e the handler cal ls panic with that value, undoing the effec t of recover and resum-ing the original state of panic. (This example does som ewhat violate our advice about not using panics for ‘‘exp ected’’ er ror s, but it prov ides a comp act illustrat ion of the mech anics.) Fr om som e condit ion s there is no recov ery. Running out of memor y, for example, causes the Go runtime to ter minate the program wit h a fat al er ror.

Exercis e 5.19: Us e panic and recover to write a function that contains no return st atement yet retur ns a non-zero value.

This page intentionally left blank ptg16091132

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

6

Me thods

Since the early 1990s, obj e ct-oriente d prog ramming (OOP) has been the dominant programming paradig m in indu str y and education, and nearly all widely used langu ages deve lop ed since then have include d supp ort for it. Go is no exception.

Although there is no univers ally accepte d definition of obj e ct-oriente d prog ramming, for our ptg16091132

purpos es, an ob jec t is simply a value or var iable that has met hods, and a metho d is a function asso ciate d with a par tic ular typ e. An obj e ct-oriente d prog ram is one that uses met hods to express the pro per ties and operat ions of each dat a st ruc ture so that clients need not access the objec t’s represent ation direc tly.

In earlier chapt ers, we have made regu lar use of met hods fro m the stand ard librar y, like the Seconds method of typ e time.Duration:

const day = 24 * time.Hour

fmt.Println(day.Seconds()) // "86400"

and we define d a met hod of our own in Sec tion 2.5, a String method for the Celsius type: func (c Celsius) String() string { return fmt.Sprintf("%g°C", c) }

In this chapt er, the firs t of two on obj e ct-oriente d prog ramming, we’ll show how to define and us e methods effec tive ly. We’ll als o cover two key princip les of obj e ct-oriente d prog ramming, encapsul ati on and comp ositi on.

6.1. Method Declarations

A met hod is declare d with a var iant of the ordinar y func tion declarat ion in which an ext ra parameter appears before the function name. The parameter attach es the function to the typ e of that parameter.

155

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

156

CHAPTER 6. METHODS

Let’s write our firs t method in a simple package for plane geometr y: gopl.io/ch6/geometry

package geometry

import "math"

type Point struct{ X, Y float64 }

// traditional function

func Distance(p, q Point) float64 {

return math.Hypot(q.X-p.X, q.Y-p.Y)

}

// same thing, but as a method of the Point type

func (p Point) Distance(q Point) float64 {

return math.Hypot(q.X-p.X, q.Y-p.Y)

}

The ext ra parameter p is cal le d the met hod’s re ceiver, a legac y from early obj e ct-oriente d langu ages that des crib ed cal ling a met hod as ‘‘sending a message to an obj e ct.’’

In Go, we don’t use a speci al name like this or self for the receiver; we cho ose receiver names just as we wou ld for any other parameter. Since the receiver name will be fre quently us ed, it’s a good ide a to choose something short and to be con sistent across met hods. A com-ptg16091132

mon choice is the firs t letter of the typ e name, like p for Point.

In a met hod cal l, the receiver argument app ears before the method name. This paral lels the de clarat ion, in which the receiver parameter appears before the method name.

p := Point{1, 2}

q := Point{4, 6}

fmt.Println(Distance(p, q)) // "5", function call fmt.Println(p.Distance(q)) // "5", method call There’s no conflic t between the two declarat ions of functions cal le d Distance ab ove . The firs t de clares a package-le vel function cal le d geometry.Distance. The secon d de clares a met hod of the typ e Point, so its name is Point.Distance.

The expression p.Distance is cal le d a selector, because it selec ts the appro priate Distance method for the receiver p of typ e Point. Selec tor s are als o us ed to selec t fields of str uct typ es, as in p.X. Since met hods and fields inhabit the same name space, declaring a met hod X on the st ruc t type Point would be ambiguous and the compi ler will rej e ct it.

Since each typ e has its own name space for met hods, we can use the name Distance for other methods so lon g as the y belong to dif ferent typ es. Let’s define a typ e Path that represents a sequence of line seg ments and give it a Distance method too.

// A Path is a journey connecting the points with straight lines.

type Path []Point

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 6.1. METHOD DECLARATIONS

157

// Distance returns the distance traveled along the path.

func (path Path) Distance() float64 {

sum := 0.0

for i := range path {

if i > 0 {

sum += path[i-1].Distance(path[i])

}

}

return sum

}

Path is a name d slice typ e, not a str uct typ e li ke Point, yet we can still define methods for it.

In allow ing met hods to be ass oci ated wit h any typ e, Go is unlike many other obj e ct-oriente d languages. It is often conv enient to define addition al beh avior s for simple typ es such as numbers, str ings, slices, maps, and som etimes even functions. Met hods may be declare d on any name d type define d in the same package, so lon g as its underly ing typ e is neither a point er nor an interface.

The two Distance methods have dif ferent typ es. They’re not rel ate d to each other at all, though Path.Distance us es Point.Distance internal ly to compute the lengt h of each segment that conne cts adjacent points.

Let’s cal l the new met hod to compute the per imeter of a rig ht triangle: ptg16091132

perim := Path{

{1, 1},

{5, 1},

{5, 4},

{1, 1},

}

fmt.Println(perim.Distance()) // "12"

In the two cal ls above to met hods named Distance, the compi ler deter mines which function to cal l based on bot h the met hod name and the typ e of the receiver. In the firs t, path[i-1]

has typ e Point so Point.Distance is cal le d; in the secon d, perim has typ e Path, so Path.Distance is cal le d.

Al l methods of a given typ e mu st have unique names, but dif ferent typ es can use the same name for a met hod, like the Distance methods for Point and Path; there’s no need to qualif y func tion names (for example, PathDistance) to dis ambigu ate. Here we see the firs t benefit to using met hods over ordinar y func tions: met hod names can be shorter. The benefit is mag nified for cal ls originating outside the package, since the y can use the shorter name an d omit the package name:

import "gopl.io/ch6/geometry"

perim := geometry.Path{{1, 1}, {5, 1}, {5, 4}, {1, 1}}

fmt.Println(geometry.PathDistance(perim)) // "12", standalone function fmt.Println(perim.Distance())

// "12", method of geometry.Path

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

158

CHAPTER 6. METHODS

6.2. Methods with a Pointer Receiver

Becaus e calling a function makes a copy of each argument value, if a function needs to update a var iable, or if an argument is so large that we wish to avoid copy ing it, we must pass the address of the var iable using a point er. The same goes for met hods that need to update the re ceiver var iable: we att ach them to the point er typ e, such as *Point.

func (p *Point) ScaleBy(factor float64) {

p.X *= factor

p.Y *= factor

}

The name of this met hod is (*Point).ScaleBy. The parenthes es are necessary ; without them, the expression wou ld be parsed as *(Point.ScaleBy).

In a realist ic program, conv ent ion dic tates that if any met hod of Point has a point er receiver, then al l methods of Point should have a point er receiver, even ones that don’t str ictly need it.

We’ve bro ken this rule for Point so that we can show bot h kind s of met hod.

Name d types (Point) and point ers to them (*Point) are the only typ es that may appear in a re ceiver declarat ion. Fur thermore, to avoid ambiguities, met hod declarat ions are not per mitte d on named typ es that are thems elves point er typ es: type P *int

ptg16091132

func (P) f() { /* ... */ } // compile error: invalid receiver type The (*Point).ScaleBy method can be cal le d by providing a *Point re ceiver, like this: r := &Point{1, 2}

r.ScaleBy(2)

fmt.Println(*r) // "{2, 4}"

or this:

p := Point{1, 2}

pptr := &p

pptr.ScaleBy(2)

fmt.Println(p) // "{2, 4}"

or this:

p := Point{1, 2}

(&p).ScaleBy(2)

fmt.Println(p) // "{2, 4}"

But the last two cas es are ungain ly. For tunately, the langu age helps us here. If the receiver p is a variab le of typ e Point but the method requires a *Point re ceiver, we can use this shorthand: p.ScaleBy(2)

and the compi ler will per for m an implicit &p on the var iable. This wor ks only for var iables, including str uct fields like p.X and array or slice elements like perim[0]. We cannot cal l a

*Point method on a non-addressable Point re ceiver, because there’s no way to obt ain the

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 6.2. METHODS WITH A POINTER RECEIVER

159

address of a temporar y value.

Point{1, 2}.ScaleBy(2) // compile error: can't take address of Point literal But we can call a Point method like Point.Distance with a *Point re ceiver, because there is a way to obt ain the value fro m the address: just load the value point ed to by the receiver. The comp iler inserts an imp licit * op erat ion for us. Thes e two function cal ls are equivalent: pptr.Distance(q)

(*pptr).Distance(q)

Let’s summar ize thes e thre e cases again, since the y are a fre quent point of confusion. In every valid met hod cal l expression, exactly one of these three statements is true.

Either the receiver argument has the same typ e as the receiver parameter, for example bot h have typ e T or bot h have typ e *T:

Point{1, 2}.Distance(q) //

Point

pptr.ScaleBy(2)

// *Point

Or the receiver argument is a variab le of typ e T and the receiver parameter has typ e *T. The comp iler implicitly takes the address of the var iable: p.ScaleBy(2) // implicit (&p)

Or the receiver argument has typ e *T and the receiver parameter has typ e T. The compi ler ptg16091132

implicitly dereferences the receiver, in other words, loads the value: pptr.Distance(q) // implicit (*pptr)

If all the methods of a named typ e T have a receiver typ e of T itself (not *T), it is safe to copy inst ances of that typ e; calling any of its met hods necessarily makes a copy. For example, time.Duration values are lib eral ly copie d, including as arguments to functions. But if any method has a point er receiver, you should avoid copy ing ins tances of T becaus e doing so may viol ate int ernal invar iants. For example, copying an ins tance of bytes.Buffer would cause the original and the copy to ali as (§2.3.2) the same underly ing array of bytes. Subsequent method cal ls wou ld have unp redic table effe cts.

6.2.1. Nil Is a Valid Receiver Value

Ju st as som e func tions allow nil point ers as arguments, so do som e methods for their receiver, especi ally if nil is a meaningf ul zero value of the typ e, as wit h maps and slices. In this simple lin ked list of int egers, nil repres ents the emp ty list:

// An IntList is a linked list of integers.

// A nil *IntList represents the empty list.

type IntList struct {

Value int

Tail *IntList

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

160

CHAPTER 6. METHODS

// Sum returns the sum of the list elements.

func (list *IntList) Sum() int {

if list == nil {

return 0

}

return list.Value + list.Tail.Sum()

}

Wh en you define a typ e whos e methods allow nil as a receiver value, it’s wor th point ing this out explicitly in its document ation comment, as we did abov e.

Here’s par t of the definition of the Values type fro m the net/url package: net/url

package url

// Values maps a string key to a list of values.

type Values map[string][]string

// Get returns the first value associated with the given key,

// or "" if there are none.

func (v Values) Get(key string) string {

if vs := v[key]; len(vs) > 0 {

return vs[0]

}

return ""

ptg16091132

}

// Add adds the value to key.

// It appends to any existing values associated with key.

func (v Values) Add(key, value string) {

v[key] = append(v[key], value)

}

It exp oses its represent ation as a map but als o prov ides met hods to simplif y access to the map, whos e values are slices of str ings—it’s a mu ltimap. Its clients can use its intr insic operator s (make, slice literals, m[key], and so on), or its met hods, or bot h, as the y prefer : gopl.io/ch6/urlvalues

m := url.Values{"lang": {"en"}} // direct construction m.Add("item", "1")

m.Add("item", "2")

fmt.Println(m.Get("lang")) // "en"

fmt.Println(m.Get("q"))

// ""

fmt.Println(m.Get("item")) // "1"

(first value)

fmt.Println(m["item"])

// "[1 2]"

(direct map access)

m = nil

fmt.Println(m.Get("item")) // ""

m.Add("item", "3")

// panic: assignment to entry in nil map

In the final cal l to Get, the nil receiver beh aves like an emp ty map. We cou ld equivalently have wr itt en it as Values(nil).Get("item")), but nil.Get("item") wi l l not compi le because

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 6.3. COMPOSING TYPES BY STRUCT EMBEDDING

161

the typ e of nil has not been deter mined. By contrast, the final cal l to Add panics as it tries to up date a nil map.

Becaus e url.Values is a map typ e and a map refers to its key/value pairs indirec tly, any up dates and delet ions that url.Values.Add makes to the map elements are visible to the caller. How ever, as wit h ordinar y func tions, any changes a met hod makes to the reference itself, like setting it to nil or mak ing it refer to a dif ferent map dat a st ruc ture, will not be reflec ted in the cal ler.

6.3. Composing Types by Struct Embedding

Consider the typ e ColoredPoint:

gopl.io/ch6/coloredpoint

import "image/color"

type Point struct{ X, Y float64 }

type ColoredPoint struct {

Point

Color color.RGBA

}

ptg16091132

We cou ld have define d ColoredPoint as a str uct of three fields, but ins tead we embedded a Point to provide the X and Y fields. As we saw in Sec tion 4.4.3, embedding lets us take a syntactic shortcut to defining a ColoredPoint that cont ains all the fields of Point, plu s some more . If we want, we can selec t the fields of ColoredPoint that were contr ibute d by the embedde d Point without mentioning Point:

var cp ColoredPoint

cp.X = 1

fmt.Println(cp.Point.X) // "1"

cp.Point.Y = 2

fmt.Println(cp.Y) // "2"

A simi lar mechanism applies to the metho ds of Point. We can cal l methods of the emb edde d Point field using a receiver of typ e ColoredPoint, even thoug h ColoredPoint has no de clare d methods:

red := color.RGBA{255, 0, 0, 255}

blue := color.RGBA{0, 0, 255, 255}

var p = ColoredPoint{Point{1, 1}, red}

var q = ColoredPoint{Point{5, 4}, blue}

fmt.Println(p.Distance(q.Point)) // "5"

p.ScaleBy(2)

q.ScaleBy(2)

fmt.Println(p.Distance(q.Point)) // "10"

The met hods of Point have been prom ote d to ColoredPoint. In this way, emb edding allows comp lex typ es with many met hods to be bui lt up by the comp ositi on of several fields, each

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

162

CHAPTER 6. METHODS

prov iding a few met hods.

Re aders fami liar wit h cl ass-b ased obj e ct-oriente d languages may be tempt ed to vie w Point as a bas e cl ass and ColoredPoint as a sub class or der ive d cl ass, or to int erpret the rel ation ship between these typ es as if a ColoredPoint ‘‘is a’’ Point. But that wou ld be a mistake . No tice the cal ls to Distance ab ove . Distance has a parameter of typ e Point, and q is not a Point, so although q do es have an emb edde d field of that typ e, we must explicitly selec t it. Att emp ting to pass q would be an error :

p.Distance(q) // compile error: cannot use q (ColoredPoint) as Point A ColoredPoint is not a Point, but it ‘‘has a’’ Point, and it has two addition al met hods Distance and ScaleBy prom ote d from Point. If you prefer to thin k in ter ms of imp lementation, the emb edde d field ins tructs the compi ler to generate addition al wrapper met hods that dele-gate to the declare d methods, equivalent to these:

func (p ColoredPoint) Distance(q Point) float64 {

return p.Point.Distance(q)

}

func (p *ColoredPoint) ScaleBy(factor float64) {

p.Point.ScaleBy(factor)

}

ptg16091132

Wh en Point.Distance is cal le d by the firs t of these wrapper met hods, its receiver value is p.Point, not p, and there is no way for the method to access the ColoredPoint in which the Point is emb edde d.

The typ e of an anony mou s field may be a pointer to a named typ e, in which cas e fields and methods are pro mot ed indirec tly fro m the point ed-to obj e ct. Adding another leve l of indire ction lets us share common str uctures and var y the rel ation ships bet ween obj e cts dynami-cally. The declarat ion of ColoredPoint below emb eds a *Point: type ColoredPoint struct {

*Point

Color color.RGBA

}

p := ColoredPoint{&Point{1, 1}, red}

q := ColoredPoint{&Point{5, 4}, blue}

fmt.Println(p.Distance(*q.Point)) // "5"

q.Point = p.Point

// p and q now share the same Point

p.ScaleBy(2)

fmt.Println(*p.Point, *q.Point) // "{2 2} {2 2}"

A str uct typ e may have more than one anony mou s field. Had we declare d ColoredPoint as type ColoredPoint struct {

Point

color.RGBA

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 6.3. COMPOSING TYPES BY STRUCT EMBEDDING

163

then a value of this typ e would have all the methods of Point, all the methods of RGBA, and any addition al met hods declare d on ColoredPoint direc tly. When the compi ler res olves a sele ctor such as p.ScaleBy to a met hod, it firs t lo oks for a direc tly declare d method named ScaleBy, then for met hods pro mot ed once fro m ColoredPoint’s emb edde d fields, then for methods pro mot ed twice fro m embedde d fields wit hin Point and RGBA, and so on. The compi ler rep orts an error if the selec tor was ambiguous because two met hods were pro mot ed fro m the same ran k.

Methods can be declare d on ly on named typ es (li ke Point) and point ers to them (*Point), but thanks to emb edding, it’s possible and som etimes useful for unname d st ruc t types to have methods too.

Here’s a nice trick to illustrate. This example shows par t of a simple cache imp lemente d using two package-le vel var iables, a mut ex (§9.2) and the map that it guard s: var (

mu sync.Mutex // guards mapping

mapping = make(map[string]string)

)

func Lookup(key string) string {

mu.Lock()

v := mapping[key]

ptg16091132

mu.Unlock()

return v

}

The version below is functionally equivalent but gro ups toget her the two rel ate d var iables in a single package-le vel var iable, cache:

var cache = struct {

sync.Mutex

mapping map[string]string

} {

mapping: make(map[string]string),

}

func Lookup(key string) string {

cache.Lock()

v := cache.mapping[key]

cache.Unlock()

return v

}

The new var iable gives more expressive names to the var iables rel ate d to the cache, and becaus e the sync.Mutex field is emb edde d within it, its Lock and Unlock methods are prom ote d to the unnamed str uct typ e, allow ing us to lock the cache with a self-explanatory sy ntax.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

164

CHAPTER 6. METHODS

6.4. Method Values and Expressions

Us ual ly we selec t and cal l a met hod in the same expression, as in p.Distance(), but it’s possible to sep arate these two operat ions. The selec tor p.Distance yields a metho d valu e, a function that binds a met hod (Point.Distance) to a specific receiver value p. This function can then be inv oke d without a receiver value; it needs only the non-receiver arguments.

p := Point{1, 2}

q := Point{4, 6}

distanceFromP := p.Distance

// method value

fmt.Println(distanceFromP(q))

// "5"

var origin Point

// {0, 0}

fmt.Println(distanceFromP(origin)) // "2.23606797749979", ;5

scaleP := p.ScaleBy // method value

scaleP(2)

// p becomes (2, 4)

scaleP(3)

//

then (6, 12)

scaleP(10)

//

then (60, 120)

Method values are useful when a package’s API cal ls for a function value, and the client’s desired beh avior for that function is to cal l a met hod on a specific receiver. For example, the func tion time.AfterFunc calls a function value after a specified del ay. This program uses it to launch the rocket r af ter 10 secon ds:

ptg16091132

type Rocket struct { /* ... */ }

func (r *Rocket) Launch() { /* ... */ }

r := new(Rocket)

time.AfterFunc(10 * time.Second, func() { r.Launch() }) The met hod value synt ax is shorter :

time.AfterFunc(10 * time.Second, r.Launch)

Relate d to the method value is the metho d expre ssi on. When cal ling a met hod, as opp osed to an ordinar y func tion, we must sup ply the receiver in a speci al way using the selec tor syntax. A method expression, writt en T.f or (*T).f where T is a typ e, yields a function value wit h a reg-ular firs t parameter tak ing the place of the receiver, so it can be cal le d in the usu al way.

p := Point{1, 2}

q := Point{4, 6}

distance := Point.Distance

// method expression

fmt.Println(distance(p, q))

// "5"

fmt.Printf("%T\n", distance) // "func(Point, Point) float64"

scale := (*Point).ScaleBy

scale(&p, 2)

fmt.Println(p)

// "{2 4}"

fmt.Printf("%T\n", scale) // "func(*Point, float64)"

Method expressions can be helpf ul when you need a value to represent a choice amon g several methods belon ging to the same typ e so that you can cal l the chosen met hod wit h many

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 6.5. EXAMPLE: BIT VECTOR TYPE

165

dif ferent receivers. In the fol low ing example, the var iable op repres ents either the addition or the subtrac tion met hod of typ e Point, and Path.TranslateBy calls it for each point in the Path:

type Point struct{ X, Y float64 }

func (p Point) Add(q Point) Point { return Point{p.X + q.X, p.Y + q.Y} }

func (p Point) Sub(q Point) Point { return Point{p.X - q.X, p.Y - q.Y} }

type Path []Point

func (path Path) TranslateBy(offset Point, add bool) {

var op func(p, q Point) Point

if add {

op = Point.Add

} else {

op = Point.Sub

}

for i := range path {

// Call either path[i].Add(offset) or path[i].Sub(offset).

path[i] = op(path[i], offset)

}

}

ptg16091132

6.5. Example: Bit Vector Type

Sets in Go are usu ally imp lemente d as a map[T]bool, where T is the element typ e. A set represent ed by a map is ver y flexible but, for cer tain pro blems, a speci alized represent ation may outp erfor m it. For example, in dom ains such as dat aflow analysis where set elements are small non-negat ive int egers, sets have many elements, and set operat ions like union and int ers ection are common, a bit vec tor is ide al.

A bit vec tor uses a slice of unsig ned int eger values or ‘‘word s,’’ each bit of which represents a possible element of the set. The set cont ains i if the i-t h bit is set. The fol low ing program demon strates a simple bit vec tor typ e with thre e methods: gopl.io/ch6/intset

// An IntSet is a set of small non-negative integers.

// Its zero value represents the empty set.

type IntSet struct {

words []uint64

}

// Has reports whether the set contains the non-negative value x.

func (s *IntSet) Has(x int) bool {

word, bit := x/64, uint(x%64)

return word < len(s.words) && s.words[word]&(1<<bit) != 0

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

166

CHAPTER 6. METHODS

// Add adds the non-negative value x to the set.

func (s *IntSet) Add(x int) {

word, bit := x/64, uint(x%64)

for word >= len(s.words) {

s.words = append(s.words, 0)

}

s.words[word] |= 1 << bit

}

// UnionWith sets s to the union of s and t.

func (s *IntSet) UnionWith(t *IntSet) {

for i, tword := range t.words {

if i < len(s.words) {

s.words[i] |= tword

} else {

s.words = append(s.words, tword)

}

}

}

Since each word has 64 bits, to locate the bit for x, we use the quotient x/64 as the word index and the remainder x%64 as the bit index wit hin that word. The UnionWith op erat ion uses the bit w ise OR operator | to compute the union 64 elements at a time. (We’ll revisit the choice of 64-bit words in Exercis e 6.5.)

ptg16091132

This imp lementation lacks many desirable features, som e of which are pos ed as exercis es below, but one is hard to live wit hout: way to print an IntSet as a str ing . Let’s give it a String method as we did wit h Celsius in Sec tion 2.5:

// String returns the set as a string of the form "{1 2 3}".

func (s *IntSet) String() string {

var buf bytes.Buffer

buf.WriteByte('{')

for i, word := range s.words {

if word == 0 {

continue

}

for j := 0; j < 64; j++ {

if word&(1<<uint(j)) != 0 {

if buf.Len() > len("{") {

buf.WriteByte(' ')

}

fmt.Fprintf(&buf, "%d", 64*i+j)

}

}

}

buf.WriteByte('}')

return buf.String()

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 6.5. EXAMPLE: BIT VECTOR TYPE

167

No tice the simi lar ity of the String method above wit h intsToString in Sec tion 3.5.4; bytes.Buffer is often used this way in String methods. The fmt package tre ats typ es with a String method speci ally so that values of complic ated typ es can displ ay thems elves in a user-friendly manner. Ins tead of print ing the raw represent ation of the value (a str uct in this cas e), fmt calls the String method. The mech anism relies on int erfaces and typ e assertions, which we’ll explain in Chapt er 7.

We can now demon strate IntSet in action:

var x, y IntSet

x.Add(1)

x.Add(144)

x.Add(9)

fmt.Println(x.String()) // "{1 9 144}"

y.Add(9)

y.Add(42)

fmt.Println(y.String()) // "{9 42}"

x.UnionWith(&y)

fmt.Println(x.String()) // "{1 9 42 144}"

fmt.Println(x.Has(9), x.Has(123)) // "true false"

A word of caution: we declare d String and Has as met hods of the point er typ e *IntSet not out of necessity, but for con sistenc y with the other two met hods, which need a point er receiver ptg16091132

becaus e they assig n to s.words. Con sequently, an IntSet valu e do es not have a String method, occ asionally leading to sur prises like this:

fmt.Println(&x)

// "{1 9 42 144}"

fmt.Println(x.String()) // "{1 9 42 144}"

fmt.Println(x)

// "{[4398046511618 0 65536]}"

In the firs t case, we print an *IntSet pointer, which does have a String method. In the second cas e, we cal l String() on an IntSet var iable; the compi ler inserts the imp licit & op er-at ion, giv ing us a point er, which has the String method. But in the third cas e, because the IntSet value does not have a String method, fmt.Println pr ints the represent ation of the st ruc t instead. It’s imp ortant not to for g et the & op erator. Mak ing String a met hod of IntSet, not *IntSet, mig ht be a good ide a, but this is a cas e-by-c ase judg ment.

Exercis e 6.1: Implement these addition al met hods: func (*IntSet) Len() int

// return the number of elements

func (*IntSet) Remove(x int)

// remove x from the set

func (*IntSet) Clear()

// remove all elements from the set

func (*IntSet) Copy() *IntSet // return a copy of the set Exercis e 6.2: Define a var iadic (*IntSet).AddAll(...int) method that allows a list of values to be adde d, such as s.AddAll(1, 2, 3).

Exercis e 6.3: (*IntSet).UnionWith comp utes the union of two sets using |, the word-p arallel bit w ise OR operator. Imp lement met hods for IntersectWith, DifferenceWith, and Sym-metricDifference for the cor responding set operat ions. (The symmetr ic dif ference of two

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

168

CHAPTER 6. METHODS

sets cont ains the elements present in one set or the other but not bot h.) Exercis e 6.4: Add a met hod Elems that retur ns a slice cont aining the elements of the set, suitable for iterat ing over wit h a range lo op.

Exercis e 6.5: The typ e of each word used by IntSet is uint64, but 64-bit arithmetic may be inefficient on a 32-bit platfor m. Modif y the program to use the uint type, which is the most efficient unsig ned int eger typ e for the platfor m. Ins tead of div iding by 64, define a con stant holding the effec tive size of uint in bits, 32 or 64. You can use the perhaps too-cle ver expression 32 << (^uint(0) >> 63) for this pur pos e.

6.6. Encapsulation

A var iable or met hod of an obj e ct is said to be encapsul ate d if it is inaccessible to clients of the objec t. Enc apsulat ion, som etimes cal le d in f o rmati on hiding, is a key asp ect of obj e ct-oriente d prog ramming.

Go has only one mech anism to control the visibilit y of names: capit alize d identifiers are exp orted fro m the package in which the y are define d, and unc apit alize d names are not. The same mech anism that limits access to members of a package als o limits access to the fields of a st ruc t or the methods of a typ e. As a con sequence, to enc apsulate an obj e ct, we must make it a ptg16091132

st ruc t.

That’s the reason the IntSet type fro m the pre vious sec tion was decl are d as a str uct typ e even though it has only a single field:

type IntSet struct {

words []uint64

}

We cou ld ins tead define IntSet as a slice typ e as fol lows, thoug h of cours e we’d have to replace each occ urrence of s.words by *s in its met hods:

type IntSet []uint64

Although this version of IntSet would be ess ent ial ly equivalent, it wou ld allow clients fro m ot her packages to read and modif y the slice direc tly. Put another way, where as the expression

*s could be used in any package, s.words may app ear only in the package that defines IntSet.

Anot her cons equence of this name-b ased mechanism is that the unit of enc apsulat ion is the package, not the typ e as in many other langu ages. The fields of a str uct typ e are visible to all co de within the same package. Whether the code app ears in a function or a met hod makes no dif ference.

Encapsulat ion provides three benefits. First, because clients cannot direc tly modif y the objec t’s var iables, one need ins pec t fe wer statements to underst and the possible values of those var iables.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 6.6. ENCAPSULATION

169

Second, hiding imp lementation det ails pre vents clients fro m dep ending on things that mig ht ch ange , which gives the desig ner gre ater fre e dom to evo l ve the imp lementation wit hout bre aking API compatibi lit y.

As an example, con sider the bytes.Buffer type. It is fre quently used to acc umulate ver y short str ings, so it is a profitable opt imizat ion to res erve a lit tle extra space in the obj e ct to avoid memor y al location in this common cas e. Since Buffer is a str uct typ e, this space takes the for m of an ext ra field of typ e [64]byte with an unc apit alize d name. When this field was adde d, because it was not exp orted, clients of Buffer outside the bytes package were unaware of any change except imp rov ed per for mance. Buffer and its Grow method are shown below, simplified for clarity :

type Buffer struct {

buf

[]byte

initial [64]byte

/* ... */

}

// Grow expands the buffer's capacity, if necessary,

// to guarantee space for another n bytes. [...]

func (b *Buffer) Grow(n int) {

if b.buf == nil {

b.buf = b.initial[:0] // use preallocated space initially

}

ptg16091132

if len(b.buf)+n > cap(b.buf) {

buf := make([]byte, b.Len(), 2*cap(b.buf) + n)

copy(buf, b.buf)

b.buf = buf

}

}

The third benefit of enc apsulat ion, and in many cas es the most imp ortant, is that it pre vents clients fro m sett ing an obj e ct’s var iables arbit rar ily. Because the obj e ct’s var iables can be set on ly by functions in the same package, the aut hor of that package can ensure that all those func tions maintain the obj e ct’s int ernal invar iants. For example, the Counter type below permits clients to increment the count er or to res et it to zero, but not to set it to som e arbit rar y value:

type Counter struct { n int }

func (c *Counter) N() int

{ return c.n }

func (c *Counter) Increment() { c.n++ }

func (c *Counter) Reset()

{ c.n = 0 }

Func tions that merely access or modif y internal values of a typ e, such as the methods of the Logger type fro m log package, below, are cal le d getters and setters. How ever, when naming a gett er met hod, we usu ally omit the Get prefix. This preference for brevit y extends to all met hods, not just field accessors, and to other redundant prefixes as wel l, such as Fetch, Find, and Lookup.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

170

CHAPTER 6. METHODS

package log

type Logger struct {

flags int

prefix string

// ...

}

func (l *Logger) Flags() int

func (l *Logger) SetFlags(flag int)

func (l *Logger) Prefix() string

func (l *Logger) SetPrefix(prefix string)

Go sty le does not forbid exp orted fields. Of cours e, on ce exp orted, a field cannot be unexported wit hout an incompatible change to the API, so the initial choice should be delib erate and should con sider the complexity of the invar iants that must be maintained, the likelihood of fut ure changes, and the quant ity of client code that wou ld be affec ted by a change .

Encapsulat ion is not always desirable. By revealing its represent ation as an int64 numb er of nanosecond s, time.Duration lets us use all the usu al arithmetic and comparison operat ions with durat ions, and even to define con stants of this typ e: const day = 24 * time.Hour

fmt.Println(day.Seconds()) // "86400"

As another example, contrast IntSet with the geometry.Path type fro m the beg inning of this ptg16091132

ch apt er. Path was define d as a slice typ e, allow ing its clients to con str uct ins tances using the slice literal synt ax, to iterate over its points using a range loop, and so on, where as these operation s are denie d to clients of IntSet.

Here’s the cruci al dif ference: geometry.Path is int rinsic ally a sequence of points, no more and no less, and we don’t fores ee adding new fields to it, so it makes sense for the geometry package to reveal that Path is a slice. In contrast, an IntSet mere ly happens to be represent ed as a

[]uint64 slice. It cou ld have been represent ed using []uint, or som ething completely dif ferent for sets that are spars e or ver y smal l, and it mig ht perhaps benefit fro m addition al features li ke an ext ra field to record the number of elements in the set. For these reasons, it makes sens e for IntSet to be opaque.

In this chapt er, we learned how to ass oci ate met hods wit h name d types, and how to cal l thos e methods. Alt hough met hods are cruci al to obj e ct-oriente d prog ramming, the y’re only half the picture. To complete it, we need interfaces, the subjec t of the next chapt er.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

7

Int erfaces

Interface typ es express generalizat ions or abstrac tions about the beh avior s of other typ es. By generalizing, int erfaces let us write functions that are more flexible and adapt able because the y are not tie d to the det ails of one par tic ular imp lementation.

Many obj e ct-oriente d languages have som e notion of int erfaces, but what makes Go’s int er-ptg16091132

faces so distinc tive is that the y are sati sfied imp licit ly. In other words, there’s no need to de clare all the int erfaces that a given concrete typ e satisfies; simply possessing the necessary methods is enoug h. This desig n lets you cre ate new int erfaces that are sat isfied by exist ing concrete typ es without chang ing the exist ing typ es, which is par tic ularly useful for typ es define d in packages that you don’t control .

In this chapt er, we’ll start by looking at the basic mechanics of int erface typ es and their values.

Alon g the way, we’ll study several important int erfaces fro m the stand ard librar y. Many Go prog rams make as much use of stand ard int erfaces as the y do of their own ones. Final ly, we’ll lo ok at ty pe ass ertions (§7.10) and ty pe switches (§7.13) and see how the y enable a different kind of generality.

7.1. Int erfaces as Contracts

Al l the typ es we’ve looke d at so far have been concre te types. A concrete typ e sp ecifies the exac t repres entation of its values and exp oses the int rinsic operat ions of that represent ation, such as arithmetic for numbers, or indexing, append, and range for slices. A concrete typ e may als o prov ide addition al beh avior s thro ugh its met hods. When you have a value of a concrete typ e, you know exac tly what it is and what you can do with it.

There is another kind of typ e in Go cal le d an interface type. An int erface is an ab strac t ty pe. It do esn’t exp ose the represent ation or int ernal str ucture of its values, or the set of basic 171

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

172

CHAPTER 7. INTERFACES

op erat ions the y supp ort; it reveals only som e of their met hods. When you have a value of an interface typ e, you know not hing about what it is; you know only what it can do, or more precis ely, what beh avior s are provide d by its met hods.

Throughout the book, we’ve been using two simi lar functions for str ing for matting: fmt.Printf, which writes the result to the stand ard out put (a file), and fmt.Sprintf, which returns the result as a string. It wou ld be unfor tunate if the hard par t, formatting the result, had to be dup lic ated because of these sup erfici al dif ferences in how the result is used. Thanks to int erfaces, it does not. Bot h of these functions are , in effe ct, wrappers aro und a third function, fmt.Fprintf, that is agnostic about what happens to the result it computes: package fmt

func Fprintf(w io.Writer, format string, args ...interface{}) (int, error) func Printf(format string, args ...interface{}) (int, error) {

return Fprintf(os.Stdout, format, args...)

}

func Sprintf(format string, args ...interface{}) string {

var buf bytes.Buffer

Fprintf(&buf, format, args...)

return buf.String()

}

The F prefix of Fprintf st and s for file and indic ates that the for matte d output should be ptg16091132

wr itt en to the file provide d as the firs t argument. In the Printf case, the argument, os.Stdout, is an *os.File. In the Sprintf case, how ever, the argument is not a file, thoug h it sup erfici ally res embles one: &buf is a point er to a memor y buffer to which bytes can be writt en.

The firs t parameter of Fprintf is not a file either. It’s an io.Writer, which is an int erface typ e with the fol low ing declarat ion:

package io

// Writer is the interface that wraps the basic Write method.

type Writer interface {

// Write writes len(p) bytes from p to the underlying data stream.

// It returns the number of bytes written from p (0 <= n <= len(p))

// and any error encountered that caused the write to stop early.

// Write must return a non-nil error if it returns n < len(p).

// Write must not modify the slice data, even temporarily.

//

// Implementations must not retain p.

Write(p []byte) (n int, err error)

}

The io.Writer interface defines the contrac t between Fprintf and its cal lers. On the one hand, the contrac t re quires that the cal ler prov ide a value of a con crete typ e li ke *os.File or

*bytes.Buffer that has a met hod cal le d Write with the appro priate sig nature and beh avior.

On the other hand, the contrac t gu arante es that Fprintf wi l l do its job given any value that satisfies the io.Writer interface. Fprintf may not assume that it is writing to a file or to

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.1. INTERFACES AS CONTRACTS

173

memory, only that it can cal l Write.

Becaus e fmt.Fprintf assumes not hing about the represent ation of the value and relies only on the beh avior s gu arante e d by the io.Writer cont rac t, we can safely pass a value of any concrete typ e that sat isfies io.Writer as the firs t argument to fmt.Fprintf. This fre e dom to su bst itute one typ e for another that sat isfies the same int erface is cal le d substitutabi lity, and is a hal lmark of obj e ct-oriente d prog ramming.

Let’s test this out using a new typ e. The Write method of the *ByteCounter type below mere ly counts the bytes writt en to it before dis carding them. (The conversion is required to make the typ es of len(p) and *c match in the += assig nment statement.) gopl.io/ch7/bytecounter

type ByteCounter int

func (c *ByteCounter) Write(p []byte) (int, error) {

*c += ByteCounter(len(p)) // convert int to ByteCounter return len(p), nil

}

Since *ByteCounter satisfies the io.Writer cont rac t, we can pass it to Fprintf, which does its str ing for matting obliv iou s to this change; the ByteCounter correc tly acc umulates the lengt h of the result.

var c ByteCounter

ptg16091132

c.Write([]byte("hello"))

fmt.Println(c) // "5", = len("hello") c = 0 // reset the counter

var name = "Dolly"

fmt.Fprintf(&c, "hello, %s", name)

fmt.Println(c) // "12", = len("hello, Dolly") Besides io.Writer, there is another interface of gre at imp ortance to the fmt package.

Fprintf and Fprintln prov ide a way for typ es to control how their values are print ed. In Section 2.5, we define d a String method for the Celsius type so that temperatures wou ld pr int as "100°C", and in Sec tion 6.5 we equip ped *IntSet with a String method so that sets would be rendered using tradition al set not ation like "{1 2 3}". Declaring a String method makes a typ e satisf y on e of the most widely used int erfaces of all, fmt.Stringer: package fmt

// The String method is used to print values passed

// as an operand to any format that accepts a string

// or to an unformatted printer such as Print.

type Stringer interface {

String() string

}

We’ll explain how the fmt package dis cov ers which values sat isf y this int erface in Sec tion 7.10.

Exercis e 7.1: Using the ide as from ByteCounter, imp lement count ers for words and for lines.

Yo u wi l l find bufio.ScanWords us eful.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

174

CHAPTER 7. INTERFACES

Exercis e 7.2: Wr ite a function CountingWriter with the sig nature below that, given an io.Writer, retur ns a new Writer that wraps the original, and a point er to an int64 var iable that at any mom ent contains the number of bytes writt en to the new Writer.

func CountingWriter(w io.Writer) (io.Writer, *int64)

Exercis e 7.3: Wr ite a String method for the *tree type in gopl.io/ch4/treesort (§4.4) that reveals the sequence of values in the tre e.

7.2. Int erface Types

An int erface typ e sp ecifies a set of met hods that a con crete typ e mu st possess to be con sidered an inst ance of that int erface.

The io.Writer type is one of the most widely used int erfaces because it provides an abstrac-tion of all the typ es to which bytes can be writt en, which includes files, memor y buffers, network conne ction s, HTTP clients, archivers, hashers, and so on. The io package defines many ot her useful int erfaces. A Reader repres ents any typ e from which you can read bytes, and a Closer is any value that you can clos e, such as a file or a net wor k connec tion. (By now you’ve prob ably not ice d the naming convent ion for many of Go’s single-met hod int erfaces.) ptg16091132

package io

type Reader interface {

Read(p []byte) (n int, err error)

}

type Closer interface {

Close() error

}

Looking far ther, we find declarat ions of new int erface typ es as combinat ions of exist ing ones.

Here are two examples:

type ReadWriter interface {

Reader

Writer

}

type ReadWriteCloser interface {

Reader

Writer

Closer

}

The syntax used above , which res embles str uct emb edding, lets us name another interface as a shorthand for writing out all of its met hods. This is cal le d embedding an interface. We cou ld have writt en io.ReadWriter without emb edding, alb eit less succinctly, like this:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.3. INTERFACE SATISFACTION

175

type ReadWriter interface {

Read(p []byte) (n int, err error)

Write(p []byte) (n int, err error)

}

or even using a mixture of the two sty les:

type ReadWriter interface {

Read(p []byte) (n int, err error)

Writer

}

Al l thre e de clarat ions have the same effec t. The order in which the methods app ear is imma-terial. All that matters is the set of met hods.

Exercis e 7.4: The strings.NewReader func tion retur ns a value that sat isfies the io.Reader interface (and others) by reading fro m its argument, a str ing . Implement a simple version of NewReader yourself, and use it to make the HTML parser (§5.2) take inp ut fro m a str ing .

Exercis e 7.5: The LimitReader func tion in the io package accepts an io.Reader r and a numb er of bytes n, and retur ns another Reader that reads fro m r but rep orts an end-of-file condit ion after n bytes. Imp lement it.

func LimitReader(r io.Reader, n int64) io.Reader

ptg16091132

7.3. Int erface Satisfaction

A typ e sati sfies an interface if it possess es all the methods the int erface requires. For example, an *os.File satisfies io.Reader, Writer, Closer, and ReadWriter. A *bytes.Buffer satisfies Reader, Writer, and ReadWriter, but does not sat isf y Closer becaus e it does not have a Close method. As a shorthand, Go programmers often say that a con crete typ e ‘‘is a’’ partic ular int erface typ e, meaning that it sat isfies the int erface. For example, a *bytes.Buffer is an io.Writer; an *os.File is an io.ReadWriter.

The assig nabilit y rule (§2.4.2) for int erfaces is ver y simple: an expression may be assig ned to an interface only if its typ e satisfies the int erface. So: var w io.Writer

w = os.Stdout

// OK: *os.File has Write method

w = new(bytes.Buffer)

// OK: *bytes.Buffer has Write method

w = time.Second

// compile error: time.Duration lacks Write method

var rwc io.ReadWriteCloser

rwc = os.Stdout

// OK: *os.File has Read, Write, Close methods

rwc = new(bytes.Buffer) // compile error: *bytes.Buffer lacks Close method This rule app lies even when the rig ht-hand side is its elf an interface: w = rwc

// OK: io.ReadWriteCloser has Write method

rwc = w

// compile error: io.Writer lacks Close method

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

176

CHAPTER 7. INTERFACES

Becaus e ReadWriter and ReadWriteCloser include all the methods of Writer, any typ e that satisfies ReadWriter or ReadWriteCloser ne cessarily sat isfies Writer.

Before we go fur ther, we should explain one subtlet y in what it means for a typ e to have a method. Recal l from Sec tion 6.2 that for each named con crete typ e T, som e of its met hods have a receiver of typ e T itself where as others require a *T pointer. Recal l also that it is legal to call a *T method on an argument of typ e T so lon g as the argument is a variab le; the compi ler implicitly takes its address. But this is mere syntactic sugar : a value of typ e T do es not possess al l the met hods that a *T pointer does, and as a result it mig ht sat isf y fe wer interfaces.

An example will make this cle ar. The String method of the IntSet type fro m Section 6.5

re quires a point er receiver, so we cannot cal l that met hod on a non-addressable IntSet value: type IntSet struct { /* ... */ }

func (*IntSet) String() string

var _ = IntSet{}.String() // compile error: String requires *IntSet receiver but we can cal l it on an IntSet var iable:

var s IntSet

var _ = s.String() // OK: s is a variable and &s has a String method Ho wever, since only *IntSet has a String method, only *IntSet satisfies the fmt.Stringer interface:

ptg16091132

var _ fmt.Stringer = &s // OK

var _ fmt.Stringer = s

// compile error: IntSet lacks String method

Section 12.8 includes a program that prints the methods of an arbit rar y value, and the godoc -analysis=type to ol (§10.7.4) displays the methods of each typ e and the rel ation ship between int erfaces and con crete typ es.

Like an env elope that wraps and con ceals the letter it holds, an int erface wraps and con ceals the con crete typ e and value that it holds. Only the methods reveale d by the int erface typ e may be cal le d, even if the con crete typ e has others:

os.Stdout.Write([]byte("hello")) // OK: *os.File has Write method os.Stdout.Close()

// OK: *os.File has Close method

var w io.Writer

w = os.Stdout

w.Write([]byte("hello")) // OK: io.Writer has Write method w.Close()

// compile error: io.Writer lacks Close method

An int erface wit h more met hods, such as io.ReadWriter, tel ls us more about the values it cont ains, and places gre ater demands on the typ es that implement it, than does an int erface with fewer met hods such as io.Reader. So what does the typ e interface{}, which has no methods at all, tel l us about the con crete typ es that sat isf y it?

That’s rig ht: not hing. This may seem useless, but in fac t the typ e interface{}, which is called the empty int erface type, is indispens able. Because the emp ty int erface typ e pl aces no demands on the typ es that sat isf y it, we can assig n any value to the emp ty int erface.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.3. INTERFACE SATISFACTION

177

var any interface{}

any = true

any = 12.34

any = "hello"

any = map[string]int{"one": 1}

any = new(bytes.Buffer)

Although it wasn’t obv iou s, we’ve been using the emp ty int erface typ e since the ver y firs t example in this book, because it is what allows functions like fmt.Println, or errorf in Section 5.7, to accept arguments of any typ e.

Of course, hav ing cre ate d an interface{} value cont aining a boole an, float, str ing , map, pointer, or any other typ e, we can do not hing direc tly to the value it holds since the int erface has no met hods. We need a way to get the value back out again. We’ll see how to do that using a ty pe ass ertion in Sec tion 7.10.

Since int erface sat isfac tion dep ends only on the methods of the two typ es invo l ved, there is no ne e d to declare the rel ation ship bet ween a con crete typ e and the int erfaces it sat isfies. That said, it is occ asionally useful to document and ass ert the rel ation ship when it is int ended but not other wis e enforce d by the program. The declarat ion below ass erts at compi le time that a value of typ e *bytes.Buffer satisfies io.Writer:

// *bytes.Buffer must satisfy io.Writer

ptg16091132

var w io.Writer = new(bytes.Buffer)

We needn’t allocate a new var iable since any value of typ e *bytes.Buffer wi l l do, even nil, which we write as (*bytes.Buffer)(nil) using an explicit conversion. And since we never intend to refer to w, we can replace it wit h the blank identifier. Toget her, these changes give us this more frugal variant:

// *bytes.Buffer must satisfy io.Writer

var _ io.Writer = (*bytes.Buffer)(nil)

No n-empt y interface typ es such as io.Writer are most often sat isfied by a point er typ e, particularly when one or more of the int erface met hods imp lies som e kind of mut ation to the re ceiver, as the Write method does. A pointer to a str uct is an esp eci ally common met hod-bear ing typ e.

But point er typ es are by no means the only typ es that sat isf y interfaces, and even int erfaces with mut ator met hods may be sat isfied by one of Go’s other reference typ es. We’ve seen examples of slice typ es with met hods (geometry.Path, §6.1) and map typ es with met hods (url.Values, §6.2.1), and later we’ll see a function typ e with met hods (http.HandlerFunc,

§7.7). Even basic typ es may sat isf y interfaces; as we saw in Sec tion 7.4, time.Duration satisfies fmt.Stringer.

A con crete typ e may sat isf y many unrel ate d interfaces. Con sider a program that organizes or sells dig itize d cult ural artifac ts li ke music, films, and books. It mig ht define the fol low ing set of con crete typ es:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

178

CHAPTER 7. INTERFACES

Album

Book

Movie

Magazine

Podcast

TVEpisode

Track

We can express each abstrac tion of int erest as an int erface. Som e prop erties are common to all ar tifac ts, such as a tit le, a cre ation date, and a list of cre ators (author s or artists).

type Artifact interface {

Title() string

Creators() []string

Created() time.Time

}

Ot her pro per ties are restr icted to cer tain typ es of artifac ts. Prop erties of the print ed word are re levant only to books and magazines, where as only mov ies and TV epi sodes have a screen resolut ion.

type Text interface {

Pages() int

Words() int

PageSize() int

ptg16091132

}

type Audio interface {

Stream() (io.ReadCloser, error)

RunningTime() time.Duration

Format() string // e.g., "MP3", "WAV"

}

type Video interface {

Stream() (io.ReadCloser, error)

RunningTime() time.Duration

Format() string // e.g., "MP4", "WMV"

Resolution() (x, y int)

}

Thes e interfaces are but one useful way to gro up rel ate d concrete typ es together and express the facets the y sh are in common. We may discov er other gro upings later. For example, if we find we need to handle Audio and Video it ems in the same way, we can define a Streamer interface to represent their common asp ects wit hout chang ing any exist ing typ e de clarat ions.

type Streamer interface {

Stream() (io.ReadCloser, error)

RunningTime() time.Duration

Format() string

}

Each gro uping of con crete typ es based on their share d behavior s can be express ed as an int erface typ e. Unlike class-b ased langu ages, in which the set of int erfaces sat isfied by a class is

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.4. PARSING FLAGS WITH FLAG.VALUE

179

explicit, in Go we can define new abstrac tions or gro upings of int erest when we need them, without modif ying the declarat ion of the con crete typ e. This is par tic ularly useful when the concrete typ e comes fro m a package writt en by a dif ferent aut hor. Of cours e, there do need to be underly ing commonalities in the con crete typ es.

7.4. Parsing Flags with flag.Value

In this sec tion, we’ll see how another stand ard int erface, flag.Value, helps us define new notation s for command-line flags. Con sider the program below, which sle eps for a specified period of time.

gopl.io/ch7/sleep

var period = flag.Duration("period", 1*time.Second, "sleep period") func main() {

flag.Parse()

fmt.Printf("Sleeping for %v...", *period)

time.Sleep(*period)

fmt.Println()

}

ptg16091132

Before it goes to sle ep it prints the time per iod. The fmt package cal ls the time.Duration’s String method to print the per iod not as a number of nanos econd s, but in a user-friendly notation:

$ go build gopl.io/ch7/sleep

$ ./sleep

Sleeping for 1s...

By defau lt, the sleep per iod is one secon d, but it can be control led through the -period command-line flag . The flag.Duration func tion cre ates a flag variable of typ e time.Duration and allows the user to specif y the durat ion in a var iety of user-friendly for mats, including the same notation printe d by the String method. This symmetr y of desig n le ads to a nice user interface.

$ ./sleep -period 50ms

Sleeping for 50ms...

$ ./sleep -period 2m30s

Sleeping for 2m30s...

$ ./sleep -period 1.5h

Sleeping for 1h30m0s...

$ ./sleep -period "1 day"

invalid value "1 day" for flag -period: time: invalid duration 1 day Becaus e durat ion-value d flags are so useful, this feature is bui lt int o the flag package, but it’s easy to define new flag not ation s for our own data typ es. We need only define a typ e that satisfies the flag.Value interface, whose declarat ion is below :

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

180

CHAPTER 7. INTERFACES

package flag

// Value is the interface to the value stored in a flag.

type Value interface {

String() string

Set(string) error

}

The String method for mats the flag’s value for use in command-line help messages; thu s ever y flag.Value is als o a fmt.Stringer. The Set method parses its str ing argument and up dates the flag value. In effec t, the Set method is the inv ers e of the String method, and it is go o d prac tice for them to use the same not ation.

Let’s define a celsiusFlag type that allows a temperature to be specified in Cel siu s, or in Fahren heit wit h an appropriate conversion. Not ice that celsiusFlag embeds a Celsius (§2.5), there by getting a String method for fre e. To sat isf y flag.Value, we need only declare the Set method:

gopl.io/ch7/tempconv

// *celsiusFlag satisfies the flag.Value interface.

type celsiusFlag struct{ Celsius }

func (f *celsiusFlag) Set(s string) error {

ptg16091132

var unit string

var value float64

fmt.Sscanf(s, "%f%s", &value, &unit) // no error check needed switch unit {

case "C", "°C":

f.Celsius = Celsius(value)

return nil

case "F", "°F":

f.Celsius = FToC(Fahrenheit(value))

return nil

}

return fmt.Errorf("invalid temperature %q", s)

}

The cal l to fmt.Sscanf pars es a floating-p oint number (value) and a str ing (unit) fro m the input s. Alt hough one must usu ally che ck Sscanf’s error result, in this cas e we don’t need to becaus e if there was a pro blem, no switch cas e wi l l match.

The CelsiusFlag func tion below wraps it all up. To the cal ler, it retur ns a point er to the Celsius field emb edde d within the celsiusFlag var iable f. The Celsius field is the var iable that will be updated by the Set method dur ing flags pro cessing. The cal l to Var adds the flag to the app lic ation’s set of command-line flags, the global var iable flag.CommandLine.

Prog rams wit h unusually complex command-line int erfaces may have several variables of this type. The cal l to Var assig ns a *celsiusFlag argument to a flag.Value parameter, causing the compi ler to che ck that *celsiusFlag has the necessary met hods.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.5. INTERFACE VALUES

181

// CelsiusFlag defines a Celsius flag with the specified name,

// default value, and usage, and returns the address of the flag variable.

// The flag argument must have a quantity and a unit, e.g., "100C".

func CelsiusFlag(name string, value Celsius, usage string) *Celsius {

f := celsiusFlag{value}

flag.CommandLine.Var(&f, name, usage)

return &f.Celsius

}

No w we can start using the new flag in our programs:

gopl.io/ch7/tempflag

var temp = tempconv.CelsiusFlag("temp", 20.0, "the temperature") func main() {

flag.Parse()

fmt.Println(*temp)

}

Here’s a typic al session:

$ go build gopl.io/ch7/tempflag

$ ./tempflag

20°C

$ ./tempflag -temp -18C

-18°C

ptg16091132

$ ./tempflag -temp 212°F

100°C

$ ./tempflag -temp 273.15K

invalid value "273.15K" for flag -temp: invalid temperature "273.15K"

Usage of ./tempflag:

-temp value

the temperature (default 20°C)

$ ./tempflag -help

Usage of ./tempflag:

-temp value

the temperature (default 20°C)

Exercis e 7.6: Add sup por t for Kelvin temperatures to tempflag.

Exercis e 7.7: Expl ain why the help message contains °C when the defau lt value of 20.0 do es not.

7.5. Int erface Values

Conceptu ally, a value of an int erface typ e, or interface value, has two components, a concrete type and a value of that typ e. These are cal le d the int erface’s dy namic type and dy namic valu e.

Fo r a statically typ ed langu age like Go, typ es are a compi le-t ime con cept, so a typ e is not a value. In our conceptu al model, a set of values cal le d ty pe des cript o rs prov ide infor mat ion

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

182

CHAPTER 7. INTERFACES

ab out each typ e, such as its name and methods. In an int erface value, the typ e comp onent is repres ente d by the appro priate typ e des crip tor.

In the four statements below, the var iable w takes on three dif ferent values. (The initial and final values are the same.)

var w io.Writer

w = os.Stdout

w = new(bytes.Buffer)

w = nil

Let’s take a clos er lo ok at the value and dynamic beh avior of w af ter each statement. The firs t st atement declares w:

var w io.Writer

In Go, var iables are always initialize d to a wel l-define d value, and int erfaces are no exception.

The zero value for an int erface has bot h its typ e and value comp onents set to nil (Figure 7.1).

ptg16091132

Figure 7.1. A nil int erface value.

An int erface value is des crib ed as nil or non-ni l based on its dynamic typ e, so this is a nil interface value. You can test whether an interface value is nil using w == nil or w != nil.

Calling any met hod of a nil int erface value causes a panic: w.Write([]byte("hello")) // panic: nil pointer dereference The secon d st atement assig ns a value of typ e *os.File to w: w = os.Stdout

This assig nment inv olves an implicit conversion fro m a con crete typ e to an int erface typ e, and is equivalent to the explicit conversion io.Writer(os.Stdout). A conv ersion of this kind, whet her explicit or imp licit, captures the typ e and the value of its operand. The int erface value’s dynamic typ e is set to the typ e des crip tor for the point er typ e *os.File, and its dy namic value holds a copy of os.Stdout, which is a pointer to the os.File var iable represent ing the stand ard out put of the pro cess (Figure 7.2).

Figure 7.2. An int erface value cont aining an *os.File pointer.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.5. INTERFACE VALUES

183

Calling the Write method on an int erface value cont aining an *os.File pointer causes the (*os.File).Write method to be cal le d. The cal l pr ints "hello".

w.Write([]byte("hello")) // "hello"

In general, we cannot know at compi le time what the dynamic typ e of an int erface value will be, so a cal l thro ugh an int erface must use dy namic dispatch. Ins tead of a direc t call, the compi ler must generate code to obt ain the address of the method named Write from the typ e des crip tor, then make an indirec t call to that address. The receiver argument for the cal l is a copy of the int erface’s dynamic value, os.Stdout. The effec t is as if we had made this cal l direc tly :

os.Stdout.Write([]byte("hello")) // "hello"

The third statement assig ns a value of typ e *bytes.Buffer to the int erface value: w = new(bytes.Buffer)

The dynamic typ e is now *bytes.Buffer and the dynamic value is a point er to the newly al locate d buffer (Figure 7.3).

ptg16091132

Figure 7.3. An int erface value cont aining a *bytes.Buffer pointer.

A cal l to the Write method uses the same mechanism as before: w.Write([]byte("hello")) // writes "hello" to the bytes.Buffer This time, the typ e des crip tor is *bytes.Buffer, so the (*bytes.Buffer).Write method is called, wit h the address of the buf fer as the value of the receiver parameter. The cal l append s

"hello" to the buf fer.

Final ly, the fourth statement assig ns nil to the int erface value: w = nil

This res ets both its comp onents to nil, restoring w to the same state as when it was decl are d, which was shown in Figure 7.1.

An int erface value can hold arbit rar ily large dynamic values. For example, the time.Time type, which represents an ins tant in time, is a str uct typ e with several unexp orted fields. If we create an int erface value fro m it,

var x interface{} = time.Now()

the result mig ht look like Figure 7.4. Con ceptu ally, the dynamic value always fits inside the interface value, no matter how large its typ e. (This is only a con ceptu al model; a realist ic implementation is quite dif ferent.)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

184

CHAPTER 7. INTERFACES

Figure 7.4. An int erface value holding a time.Time st ruc t.

Interface values may be compare d using == and !=. Two int erface values are equ al if bot h are ni l, or if their dynamic typ es are identical and their dynamic values are equ al according to the usual beh avior of == for that typ e. Because int erface values are comparable, the y may be used as the keys of a map or as the operand of a switch statement.

Ho wever, if two int erface values are compare d and have the same dynamic typ e, but that typ e is not comparable (a slice, for ins tance), then the comparison fai ls wit h a panic: var x interface{} = []int{1, 2, 3}

fmt.Println(x == x) // panic: comparing uncomparable type []int In this respect, int erface typ es are unu sual. Other typ es are eit her safely comparable (like basic typ es and point ers) or not comparable at all (li ke slices, maps, and functions), but when ptg16091132

comp aring int erface values or aggregate typ es that cont ain interface values, we must be aware of the pot ent ial for a panic. A simi lar risk exists when using int erfaces as map keys or switch op erands. Only compare int erface values if you are cer tain that the y cont ain dynamic values of comparable typ es.

Wh en handling error s, or dur ing debug ging, it is often helpf ul to rep ort the dynamic typ e of an interface value. For that, we use the fmt package’s %T verb: var w io.Writer

fmt.Printf("%T\n", w) // "<nil>"

w = os.Stdout

fmt.Printf("%T\n", w) // "*os.File"

w = new(bytes.Buffer)

fmt.Printf("%T\n", w) // "*bytes.Buffer"

Internal ly, fmt us es reflec tion to obt ain the name of the int erface’s dynamic typ e. We’ll look at reflec tion in Chapt er 12.

7.5.1. Caveat: An Int erface Containing a Nil Pointer Is Non-Nil A nil int erface value, which contains no value at all, is not the same as an int erface value containing a point er that happens to be nil. This subtle distinc tion cre ates a trap into which every Go programmer has stumb led.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.5. INTERFACE VALUES

185

Consider the program below. Wit h debug set to true, the main function col lec ts the out put of the function f in a bytes.Buffer.

const debug = true

func main() {

var buf *bytes.Buffer

if debug {

buf = new(bytes.Buffer) // enable collection of output

}

f(buf) // NOTE: subtly incorrect!

if debug {

// ...use buf...

}

}

// If out is non-nil, output will be written to it.

func f(out io.Writer) {

// ...do something...

if out != nil {

out.Write([]byte("done!\n"))

}

}

We mig ht exp ect that chang ing debug to false would dis able the col lec tion of the out put, but ptg16091132

in fact it causes the program to panic dur ing the out.Write call: if out != nil {

out.Write([]byte("done!\n")) // panic: nil pointer dereference

}

Wh en main calls f, it assig ns a nil point er of typ e *bytes.Buffer to the out parameter, so the dy namic value of out is nil. How ever, its dynamic typ e is *bytes.Buffer, meaning that out is a non-ni l interface cont aining a nil point er value (Figure 7.5), so the defensive che ck out != nil is still true.

Figure 7.5. A non-ni l interface cont aining a nil point er.

As before, the dynamic disp atch mechanism deter mines that (*bytes.Buffer).Write mu st be cal le d but this time wit h a receiver value that is nil. For som e types, such as *os.File, nil is a valid receiver (§6.2.1), but *bytes.Buffer is not among them. The met hod is cal le d, but it panics as it tries to access the buf fer.

The pro blem is that alt hough a nil *bytes.Buffer pointer has the methods needed to sat isf y the int erface, it doesn’t sat isf y the behavioral re quirements of the int erface. In par tic ular, the

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

186

CHAPTER 7. INTERFACES

call violates the imp licit pre con dit ion of (*bytes.Buffer).Write that its receiver is not nil, so assig ning the nil point er to the int erface was a mist ake . The solut ion is to change the typ e of buf in main to io.Writer, there by avoiding the assig nment of the dysf unctional value to the int erface in the firs t pl ace:

var buf io.Writer

if debug {

buf = new(bytes.Buffer) // enable collection of output

}

f(buf) // OK

No w that we’ve cov ere d the mechanics of int erface values, let’s take a look at som e more important int erfaces fro m Go’s stand ard librar y. In the next three sec tion s, we’ll see how int erfaces are used for sor ting, web ser ving, and error handling.

7.6. Sorting with sort.Interface

Like str ing for matting , sorting is a fre quently used operat ion in many programs. Alt hough a minimal Quicks ort can be writt en in about 15 lines, a robust imp lementation is much lon g er, and it is not the kind of code we should wish to write ane w or copy each time we need it.

ptg16091132

Fo rtunate ly, the sort package provides in-place sor ting of any sequence according to any order ing function. Its design is rat her unusual. In many langu ages, the sor ting algor it hm is asso ciate d with the sequence dat a type, whi le the order ing function is ass oci ated wit h the typ e of the elements. By contrast, Go’s sort.Sort func tion assumes not hing about the representation of eit her the sequence or its elements. Ins tead, it uses an int erface, sort.Interface, to sp ecif y the contrac t between the gener ic sort algor it hm and each sequence typ e that may be sorted. An imp lementation of this int erface deter mines bot h the con crete represent ation of the sequence, which is often a slice, and the desired order ing of its elements.

An in-place sor t algor it hm needs three things—the lengt h of the sequence, a means of comparing two elements, and a way to swap two elements—so the y are the three met hods of sort.Interface:

package sort

type Interface interface {

Len() int

Less(i, j int) bool // i, j are indices of sequence elements Swap(i, j int)

}

To sor t any sequence, we need to define a typ e that implements these three met hods, then apply sort.Sort to an ins tance of that typ e. As perhaps the simplest example, con sider sorting a slice of str ings. The new typ e StringSlice and its Len, Less, and Swap methods are shown below.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.6. SORTING WITH SORT.INTERFACE

187

type StringSlice []string

func (p StringSlice) Len() int

{ return len(p) }

func (p StringSlice) Less(i, j int) bool { return p[i] < p[j] }

func (p StringSlice) Swap(i, j int)

{ p[i], p[j] = p[j], p[i] }

No w we can sor t a slice of str ings, names, by converting the slice to a StringSlice li ke this: sort.Sort(StringSlice(names))

The conversion yields a slice value wit h the same lengt h, capacity, and underly ing array as names but wit h a typ e that has the three met hods required for sor ting.

Sorting a slice of str ings is so common that the sort package provides the StringSlice type, as wel l as a function cal le d Strings so that the cal l ab ove can be simplified to sort.Strings(names).

The technique here is easi ly adapt ed to other sor t orders, for ins tance, to ignore capit alizat ion or speci al ch arac ters. (The Go program that sor ts index ter ms and page numbers for this book does this, wit h ext ra log ic for Rom an numerals.) For more complic ated sor ting, we use the same ide a, but wit h more complic ated dat a st ruc tures or more complic ated imp lementation s of the sort.Interface methods.

Our running example for sor ting will be a music playlist, displaye d as a table. Each track is a single row, and each column is an att ribut e of that track, like artist, tit le, and running time.

ptg16091132

Im agine that a graphic al us er int erface presents the table, and that clicking the head of a column causes the playlist to be sor ted by that att ribut e; clicking the same column head again re verses the order. Let’s look at what mig ht happen in respons e to each click.

The var iable tracks below contains a playlist. (One of the aut hor s ap olog izes for the other author’s music al tastes.) Each element is indirec t, a point er to a Track. Alt hough the code below wou ld wor k if we store d the Tracks direc tly, the sor t func tion will swap many pairs of elements, so it will run faster if each element is a point er, which is a single machine word, instead of an ent ire Track, which mig ht be eig ht words or more.

gopl.io/ch7/sorting

type Track struct {

Title string

Artist string

Album string

Year

int

Length time.Duration

}

var tracks = []*Track{

{"Go", "Delilah", "From the Roots Up", 2012, length("3m38s")},

{"Go", "Moby", "Moby", 1992, length("3m37s")},

{"Go Ahead", "Alicia Keys", "As I Am", 2007, length("4m36s")},

{"Ready 2 Go", "Martin Solveig", "Smash", 2011, length("4m24s")},

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

188

CHAPTER 7. INTERFACES

func length(s string) time.Duration {

d, err := time.ParseDuration(s)

if err != nil {

panic(s)

}

return d

}

The printTracks func tion prints the playlist as a table. A graphic al display wou ld be nicer, but this lit tle routine uses the text/tabwriter package to pro duce a table whose columns are ne atly alig ned and padde d as shown below. Obs erve that *tabwriter.Writer satisfies io.Writer. It col lec ts each pie ce of dat a wr itt en to it; its Flush method for mats the ent ire table and writes it to os.Stdout.

func printTracks(tracks []*Track) {

const format = "%v\t%v\t%v\t%v\t%v\t\n"

tw := new(tabwriter.Writer).Init(os.Stdout, 0, 8, 2, ' ', 0) fmt.Fprintf(tw, format, "Title", "Artist", "Album", "Year", "Length") fmt.Fprintf(tw, format, "-----", "------", "-----", "----", "------") for _, t := range tracks {

fmt.Fprintf(tw, format, t.Title, t.Artist, t.Album, t.Year, t.Length)

}

tw.Flush() // calculate column widths and print table

}

ptg16091132

To sor t the playlist by the Artist field, we define a new slice typ e with the necessary Len, Less, and Swap methods, analogou s to what we did for StringSlice.

type byArtist []*Track

func (x byArtist) Len() int

{ return len(x) }

func (x byArtist) Less(i, j int) bool { return x[i].Artist < x[j].Artist }

func (x byArtist) Swap(i, j int)

{ x[i], x[j] = x[j], x[i] }

To cal l the gener ic sort routine, we must firs t conv ert tracks to the new typ e, byArtist, that defines the order :

sort.Sort(byArtist(tracks))

Af ter sor ting the slice by artist, the out put fro m printTracks is Title

Artist

Album

Year

Length

-----

------

-----

----

------

Go Ahead

Alicia Keys

As I Am

2007 4m36s

Go

Delilah

From the Roots Up

2012 3m38s

Ready 2 Go

Martin Solveig

Smash

2011

4m24s

Go

Moby

Moby

1992

3m37s

If the user requests ‘‘sort by artist’’ a secon d time, we’ll sor t the tracks in reverse. We needn’t define a new typ e byReverseArtist with an inv erted Less method, how ever, since the sort package provides a Reverse func tion that transfor ms any sor t order to its invers e.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.6. SORTING WITH SORT.INTERFACE

189

sort.Sort(sort.Reverse(byArtist(tracks)))

Af ter reverse-s orting the slice by artist, the out put fro m printTracks is Title

Artist

Album

Year

Length

-----

------

-----

----

------

Go

Moby

Moby

1992

3m37s

Ready 2 Go

Martin Solveig

Smash

2011

4m24s

Go

Delilah

From the Roots Up

2012 3m38s

Go Ahead

Alicia Keys

As I Am

2007 4m36s

The sort.Reverse func tion des erves a clos er lo ok since it uses composition (§6.3), which is an important ide a. The sort package defines an unexp orted typ e reverse, which is a str uct that embeds a sort.Interface. The Less method for reverse calls the Less method of the embedde d sort.Interface value, but wit h the indices flip ped, reversing the order of the sor t resu lts.

package sort

type reverse struct{ Interface } // that is, sort.Interface func (r reverse) Less(i, j int) bool { return r.Interface.Less(j, i) }

func Reverse(data Interface) Interface { return reverse{data} }

Len and Swap, the other two met hods of reverse, are imp licitly provide d by the original ptg16091132

sort.Interface value because it is an emb edde d field. The exp orted function Reverse returns an ins tance of the reverse type that contains the original sort.Interface value.

To sor t by a dif ferent column, we must define a new typ e, such as byYear: type byYear []*Track

func (x byYear) Len() int

{ return len(x) }

func (x byYear) Less(i, j int) bool { return x[i].Year < x[j].Year }

func (x byYear) Swap(i, j int)

{ x[i], x[j] = x[j], x[i] }

Af ter sor ting tracks by year using sort.Sort(byYear(tracks)), printTracks shows a chro nolog ical list ing:

Title

Artist

Album

Year

Length

-----

------

-----

----

------

Go

Moby

Moby

1992

3m37s

Go Ahead

Alicia Keys

As I Am

2007 4m36s

Ready 2 Go

Martin Solveig

Smash

2011

4m24s

Go

Delilah

From the Roots Up

2012 3m38s

Fo r ever y slice element typ e and every order ing function we need, we declare a new imp lement ation of sort.Interface. As you can see, the Len and Swap methods have identical definition s for all slice typ es. In the next example, the con crete typ e customSort combines a slice with a func tion, letting us define a new sor t order by writing only the comparison function.

In cidentally, the con crete typ es that implement sort.Interface are not always slices; customSort is a str uct typ e.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

190

CHAPTER 7. INTERFACES

type customSort struct {

t

[]*Track

less func(x, y *Track) bool

}

func (x customSort) Len() int

{ return len(x.t) }

func (x customSort) Less(i, j int) bool { return x.less(x.t[i], x.t[j]) }

func (x customSort) Swap(i, j int)

{ x.t[i], x.t[j] = x.t[j], x.t[i] }

Let’s define a multi-t ier order ing function whose primary sor t ke y is the Title, whose second ary key is the Year, and whose ter tiary key is the running time, Length. Here’s the cal l to Sort using an anony mou s order ing function:

sort.Sort(customSort{tracks, func(x, y *Track) bool {

if x.Title != y.Title {

return x.Title < y.Title

}

if x.Year != y.Year {

return x.Year < y.Year

}

if x.Length != y.Length {

return x.Length < y.Length

}

return false

ptg16091132

}})

And here’s the result. Not ice that the tie bet ween the two tracks tit led ‘‘Go’’ is bro ken in favor of the older one.

Title

Artist

Album

Year

Length

-----

------

-----

----

------

Go

Moby

Moby

1992

3m37s

Go

Delilah

From the Roots Up

2012 3m38s

Go Ahead

Alicia Keys

As I Am

2007 4m36s

Ready 2 Go

Martin Solveig

Smash

2011

4m24s

Although sor ting a sequence of lengt h n re quires O(n log n) comparison operat ions, testing whet her a sequence is already sor ted requires at most n−1 comparisons. The IsSorted function fro m the sort package che cks this for us. Like sort.Sort, it abstrac ts both the sequence and its order ing function using sort.Interface, but it never cal ls the Swap method: This co de demon strates the IntsAreSorted and Ints func tions and the IntSlice type: values := []int{3, 1, 4, 1}

fmt.Println(sort.IntsAreSorted(values)) // "false"

sort.Ints(values)

fmt.Println(values)

// "[1 1 3 4]"

fmt.Println(sort.IntsAreSorted(values)) // "true"

sort.Sort(sort.Reverse(sort.IntSlice(values)))

fmt.Println(values)

// "[4 3 1 1]"

fmt.Println(sort.IntsAreSorted(values)) // "false"

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.7. THE HTTP.HANDLER INTERFACE

191

Fo r conv enience, the sort package provides versions of its functions and typ es sp eci alized for

[]int, []string, and []float64 using their natural order ings. For other typ es, such as

[]int64 or []uint, we’re on our own, thoug h the pat h is short.

Exercis e 7.8: Many GUIs provide a table widget wit h a stateful multi-t ier sort: the primary sort key is the most recently clicke d column head, the secon dar y sort key is the secon d-most re cently clicke d column head, and so on. Define an imp lementation of sort.Interface for us e by such a table. Compare that appro ach wit h repeated sor ting using sort.Stable.

Exercis e 7.9: Us e the html/template package (§4.6) to replace printTracks with a func tion that displays the tracks as an HTML table. Use the solut ion to the pre vious exercis e to arrange that each click on a column head makes an HTTP request to sor t the table.

Exercis e 7.10: The sort.Interface type can be adapt ed to other uses. Write a function IsPalindrome(s sort.Interface) bool that rep orts whether the sequence s is a palindrom e, in other words, reversing the sequence wou ld not change it. Assume that the elements at indices i and j are equ al if !s.Less(i, j) && !s.Less(j, i).

7.7. The http.Handler Int erface

In Chapt er 1, we saw a glimps e of how to use the net/http package to imp lement web clients ptg16091132

(§1.5) and ser vers (§1.7). In this sec tion, we’ll look more clos ely at the ser ver API, whose foundation is the http.Handler interface:

net/http

package http

type Handler interface {

ServeHTTP(w ResponseWriter, r *Request)

}

func ListenAndServe(address string, h Handler) error

The ListenAndServe func tion requires a ser ver address, such as "localhost:8000", and an inst ance of the Handler interface to which all requests should be dispatch ed. It runs forever, or unt i l the ser ver fails (or fai ls to start) wit h an erro r, always non-ni l, which it retur ns.

Im agine an e-commerce site wit h a dat abas e mapping the items for sale to their prices in dollars. The program below shows the simplest imaginable imp lementation. It model s the inv en-tory as a map typ e, database, to which we’ve att ach ed a ServeHTTP method so that it sat isfies the http.Handler interface. The handler ranges over the map and prints the items.

gopl.io/ch7/http1

func main() {

db := database{"shoes": 50, "socks": 5}

log.Fatal(http.ListenAndServe("localhost:8000", db))

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

192

CHAPTER 7. INTERFACES

type dollars float32

func (d dollars) String() string { return fmt.Sprintf("$%.2f", d) }

type database map[string]dollars

func (db database) ServeHTTP(w http.ResponseWriter, req *http.Request) {

for item, price := range db {

fmt.Fprintf(w, "%s: %s\n", item, price)

}

}

If we start the ser ver,

$ go build gopl.io/ch7/http1

$ ./http1 &

then connec t to it wit h the fetch prog ram from Sec tion 1.5 (or a web brows er if you prefer), we get the fol low ing out put:

$ go build gopl.io/ch1/fetch

$ ./fetch http://localhost:8000

shoes: $50.00

socks: $5.00

So far, the ser ver can only list its entire inv ent ory and will do this for every request, regardless ptg16091132

of URL. A more realist ic ser ver defines multiple dif ferent URLs, each trigger ing a dif ferent behavior. Let’s cal l the exist ing one /list and add anot her one cal le d /price that rep orts the pr ice of a single item, specified as a request parameter like /price?item=socks.

gopl.io/ch7/http2

func (db database) ServeHTTP(w http.ResponseWriter, req *http.Request) {

switch req.URL.Path {

case "/list":

for item, price := range db {

fmt.Fprintf(w, "%s: %s\n", item, price)

}

case "/price":

item := req.URL.Query().Get("item")

price, ok := db[item]

if !ok {

w.WriteHeader(http.StatusNotFound) // 404

fmt.Fprintf(w, "no such item: %q\n", item)

return

}

fmt.Fprintf(w, "%s\n", price)

default:

w.WriteHeader(http.StatusNotFound) // 404

fmt.Fprintf(w, "no such page: %s\n", req.URL)

}

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.7. THE HTTP.HANDLER INTERFACE

193

No w the handler decides what log ic to exe cut e based on the pat h comp onent of the URL, req.URL.Path. If the handler doesn’t recog nize the pat h, it rep orts an HTTP error to the client by cal ling w.WriteHeader(http.StatusNotFound); this must be don e before writing any text to w. (In cidentally, http.ResponseWriter is another interface. It aug ments io.Writer with met hods for sending HTTP respons e he aders.) Equivalently, we cou ld use the http.Error ut i lit y func tion:

msg := fmt.Sprintf("no such page: %s\n", req.URL) http.Error(w, msg, http.StatusNotFound) // 404

The cas e for /price calls the URL’s Query method to parse the HTTP request parameters as a map, or more pre cis ely, a multimap of typ e url.Values (§6.2.1) fro m the net/url package. It then find s the firs t item parameter and looks up its price. If the item wasn’t found, it rep orts an erro r.

Here’s an example session wit h the new server :

$ go build gopl.io/ch7/http2

$ go build gopl.io/ch1/fetch

$ ./http2 &

$ ./fetch http://localhost:8000/list

shoes: $50.00

socks: $5.00

ptg16091132

$ ./fetch http://localhost:8000/price?item=socks

$5.00

$ ./fetch http://localhost:8000/price?item=shoes

$50.00

$ ./fetch http://localhost:8000/price?item=hat

no such item: "hat"

$ ./fetch http://localhost:8000/help

no such page: /help

Obviou sly we cou ld keep adding cas es to ServeHTTP, but in a realist ic app lic ation, it’s convenient to define the log ic for each cas e in a sep arate function or met hod. Fur thermore, re lated URLs may need simi lar logic; several image files may have URLs of the for m

/images/*.png, for ins tance. For these reasons, net/http prov ides ServeMux, a re quest mu ltipl exer, to simplif y the ass oci ation bet ween URLs and handlers. A ServeMux ag gregates a collec tion of http.Handlers int o a single http.Handler. Again, we see that dif ferent typ es satisf ying the same int erface are substitutabl e: the web ser ver can disp atch requests to any http.Handler, regardless of which con crete typ e is behind it.

Fo r a more complex applic ation, several ServeMuxes may be composed to handle more intr icate dispatching requirements. Go doesn’t have a canonic al we b framewor k analogou s to Ru by’s Rai ls or Pyt hon’s Django. This is not to say that such framewor ks don’t exist, but the building blo cks in Go’s stand ard librar y are flexible enoug h that framewor ks are often unnecessary. Fur thermore, alt hough framewor ks are convenient in the early phas es of a proj e ct, their addition al complexity can make lon g er-term maintenance harder.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

194

CHAPTER 7. INTERFACES

In the program below, we cre ate a ServeMux and use it to ass oci ate the URLs wit h the corresp onding handlers for the /list and /price op erat ions, which have been split int o separate methods. We then use the ServeMux as the main handler in the cal l to ListenAndServe.

gopl.io/ch7/http3

func main() {

db := database{"shoes": 50, "socks": 5}

mux := http.NewServeMux()

mux.Handle("/list", http.HandlerFunc(db.list)) mux.Handle("/price", http.HandlerFunc(db.price)) log.Fatal(http.ListenAndServe("localhost:8000", mux))

}

type database map[string]dollars

func (db database) list(w http.ResponseWriter, req *http.Request) {

for item, price := range db {

fmt.Fprintf(w, "%s: %s\n", item, price)

}

}

func (db database) price(w http.ResponseWriter, req *http.Request) {

item := req.URL.Query().Get("item")

price, ok := db[item]

if !ok {

ptg16091132

w.WriteHeader(http.StatusNotFound) // 404

fmt.Fprintf(w, "no such item: %q\n", item)

return

}

fmt.Fprintf(w, "%s\n", price)

}

Let’s focus on the two cal ls to mux.Handle that reg ister the handlers. In the firs t on e, db.list is a met hod value (§6.4), that is, a value of typ e

func(w http.ResponseWriter, req *http.Request)

that, when cal le d, inv okes the database.list method wit h the receiver value db. So db.list is a function that imp lements handler-li ke beh avior, but since it has no met hods, it doesn’t satisfy the http.Handler interface and can’t be passed direc tly to mux.Handle.

The expression http.HandlerFunc(db.list) is a conversion, not a function cal l, since http.HandlerFunc is a typ e. It has the fol low ing definition: net/http

package http

type HandlerFunc func(w ResponseWriter, r *Request)

func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {

f(w, r)

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.7. THE HTTP.HANDLER INTERFACE

195

HandlerFunc demon strates som e unusual features of Go’s int erface mechanism. It is a function typ e that has met hods and sat isfies an interface, http.Handler. The beh avior of its ServeHTTP method is to cal l the underly ing function. HandlerFunc is thu s an adapter that lets a function value sat isf y an interface, where the function and the int erface’s sole met hod have the same sig nature. In effec t, this trick lets a single typ e such as database satisf y the http.Handler interface several different ways: once through its list method, once through its price method, and so on.

Becaus e register ing a handler this way is so common, ServeMux has a convenience met hod called HandleFunc that does it for us, so we can simplif y the handler reg ist rat ion code to this: gopl.io/ch7/http3a

mux.HandleFunc("/list", db.list)

mux.HandleFunc("/price", db.price)

It’s easy to see fro m the code above how one wou ld con str uct a program in which there are two dif ferent web ser vers, listening on dif ferent por ts, defining dif ferent URLs, and dispatching to dif ferent handlers. We wou ld just con str uct another ServeMux and make another call to ListenAndServe, perhaps conc urrently. But in most programs, one web ser ver is plenty. Als o, it’s typic al to define HTTP handlers across many files of an app lic ation, and it would be a nuisance if the y al l had to be explicitly reg istered wit h the app lic ation’s ServeMux inst ance.

ptg16091132

So, for convenience, net/http prov ides a global ServeMux inst ance cal le d DefaultServeMux and package-le vel functions cal le d http.Handle and http.HandleFunc. To use DefaultServeMux as the ser ver’s main handler, we needn’t pass it to ListenAndServe; nil wi l l do.

The ser ver’s main function can then be simplified to

gopl.io/ch7/http4

func main() {

db := database{"shoes": 50, "socks": 5}

http.HandleFunc("/list", db.list)

http.HandleFunc("/price", db.price)

log.Fatal(http.ListenAndServe("localhost:8000", nil))

}

Final ly, an imp ortant reminder : as we mention ed in Sec tion 1.7, the web ser ver invo kes each hand ler in a new goroutine, so handlers must take pre caution s such as locki n g when accessing var iables that other goroutines, including other requests to the same handler, may be accessing. We’ll tal k ab out con cur rency in the next two chapt ers.

Exercis e 7.11: Add addition al handlers so that clients can create, read, update, and delete database ent ries. For example, a request of the for m /update?item=socks&price=6 wi l l up date the price of an item in the inv ent ory and rep ort an error if the item does not exist or if the price is invalid. (Warning: this change int roduces conc urrent var iable updates.) Exercis e 7.12: Change the handler for /list to print its out put as an HTML table, not text.

Yo u may find the html/template package (§4.6) useful.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

196

CHAPTER 7. INTERFACES

7.8. The error Int erface

Since the beg inning of this book, we’ve been using and cre ating values of the mysteriou s predeclare d error type wit hout explaining what it really is. In fac t, it’s just an int erface typ e with a single met hod that retur ns an error message:

type error interface {

Error() string

}

The simplest way to cre ate an error is by cal ling errors.New, which retur ns a new error for a given erro r mess age . The ent ire errors package is only four lines long: package errors

func New(text string) error { return &errorString{text} }

type errorString struct { text string }

func (e *errorString) Error() string { return e.text }

The underly ing typ e of errorString is a str uct, not a str ing , to pro tec t its represent ation fro m inadvertent (or premedit ate d) updates. And the reason that the point er typ e *errorString, ptg16091132

not errorString alone, sat isfies the error interface is so that every cal l to New al locates a distinc t error inst ance that is equ al to no other. We wou ld not want a distinguishe d er ror such as io.EOF to compare equ al to one that merely happene d to have the same message .

fmt.Println(errors.New("EOF") == errors.New("EOF")) // "false"

Calls to errors.New are rel ative ly inf requent because there’s a convenient wrapper function, fmt.Errorf, that does str ing for matting too. We used it several times in Chapt er 5.

package fmt

import "errors"

func Errorf(format string, args ...interface{}) error {

return errors.New(Sprintf(format, args...))

}

Although *errorString may be the simplest typ e of error, it is far fro m the only one. For example, the syscall package provides Go’s low-le vel system cal l API. On many platfor ms, it defines a numeric typ e Errno that sat isfies error, and on Unix platfor ms, Errno’s Error method does a looku p in a table of str ings, as shown below : package syscall

type Errno uintptr // operating system error code

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.9. EXAMPLE: EXPRESSION EVALUATOR

197

var errors = [...]string{

1:

"operation not permitted",

// EPERM

2:

"no such file or directory", // ENOENT

3:

"no such process",

// ESRCH

// ...

}

func (e Errno) Error() string {

if 0 <= int(e) && int(e) < len(errors) {

return errors[e]

}

return fmt.Sprintf("errno %d", e)

}

The fol low ing statement cre ates an int erface value holding the Errno value 2, sig nifying the POSIX ENOENT condit ion:

var err error = syscall.Errno(2)

fmt.Println(err.Error()) // "no such file or directory"

fmt.Println(err)

// "no such file or directory"

The value of err is shown graphic ally in Figure 7.6.

ptg16091132

Figure 7.6. An int erface value holding a syscall.Errno integer.

Errno is an efficient represent ation of system cal l er ror s draw n from a finite set, and it sat isfies the stand ard error interface. We’ll see other typ es that sat isf y this int erface in Sec tion 7.11.

7.9. Example: Expression Evaluator

In this sec tion, we’ll bui ld an evaluator for simple arithmetic expressions. We’ll use an int erface, Expr, to represent any expression in this langu age. For now, this int erface needs no methods, but we’ll add som e later.

// An Expr is an arithmetic expression.

type Expr interface{}

Our expression langu age con sists of floating-p oint lit erals; the binar y op erator s +, -, *, and /; the unary operator s -x and +x; function cal ls pow(x,y), sin(x), and sqrt(x); var iables such as x and pi; and of cours e parent hes es and stand ard operator precedence. All values are of type float64. Here are som e example expressions:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

198

CHAPTER 7. INTERFACES

sqrt(A / pi)

pow(x, 3) + pow(y, 3)

(F - 32) * 5 / 9

The five con crete typ es below represent par tic ular kinds of expression. A Var repres ents a reference to a var iable. (We’ll soon see why it is exp orted.) A literal repres ents a floating-point con stant. The unary and binary types represent operator expressions wit h on e or two op erands, which can be any kind of Expr. A call repres ents a function cal l; we’ll restr ict its fn field to pow, sin, or sqrt.

gopl.io/ch7/eval

// A Var identifies a variable, e.g., x.

type Var string

// A literal is a numeric constant, e.g., 3.141.

type literal float64

// A unary represents a unary operator expression, e.g., -x.

type unary struct {

op rune // one of '+', '-'

x

Expr

}

// A binary represents a binary operator expression, e.g., x+y.

type binary struct {

op

rune // one of '+', '-', '*', '/'

ptg16091132

x, y Expr

}

// A call represents a function call expression, e.g., sin(x).

type call struct {

fn

string // one of "pow", "sin", "sqrt"

args []Expr

}

To evaluate an expression containing var iables, we’ll need an enviro nment that maps variable names to values:

type Env map[Var]float64

We’ll als o ne e d each kind of expression to define an Eval method that retur ns the expression’s value in a given enviro nment. Since every expression must provide this met hod, we add it to the Expr interface. The package exp orts only the typ es Expr, Env, and Var; clients can use the evaluator wit hout access to the other expression typ es.

type Expr interface {

// Eval returns the value of this Expr in the environment env.

Eval(env Env) float64

}

The con crete Eval methods are shown below. The method for Var perfor ms an env iro nment lo oku p, which retur ns zero if the var iable is not define d, and the method for literal simply returns the lit eral value.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.9. EXAMPLE: EXPRESSION EVALUATOR

199

func (v Var) Eval(env Env) float64 {

return env[v]

}

func (l literal) Eval(_ Env) float64 {

return float64(l)

}

The Eval methods for unary and binary re cursive ly evaluate their operands, then app l y the op erat ion op to them. We don’t con sider divisions by zero or infinity to be error s, since the y produce a result, albeit non-finite. Final ly, the method for call evaluates the arguments to the pow, sin, or sqrt func tion, then cal ls the cor responding function in the math package.

func (u unary) Eval(env Env) float64 {

switch u.op {

case '+':

return +u.x.Eval(env)

case '-':

return -u.x.Eval(env)

}

panic(fmt.Sprintf("unsupported unary operator: %q", u.op))

}

func (b binary) Eval(env Env) float64 {

switch b.op {

case '+':

ptg16091132

return b.x.Eval(env) + b.y.Eval(env)

case '-':

return b.x.Eval(env) - b.y.Eval(env)

case '*':

return b.x.Eval(env) * b.y.Eval(env)

case '/':

return b.x.Eval(env) / b.y.Eval(env)

}

panic(fmt.Sprintf("unsupported binary operator: %q", b.op))

}

func (c call) Eval(env Env) float64 {

switch c.fn {

case "pow":

return math.Pow(c.args[0].Eval(env), c.args[1].Eval(env)) case "sin":

return math.Sin(c.args[0].Eval(env))

case "sqrt":

return math.Sqrt(c.args[0].Eval(env))

}

panic(fmt.Sprintf("unsupported function call: %s", c.fn))

}

Several of these met hods can fail. For example, a call expression cou ld have an unknown func tion or the wro ng number of arguments. It’s als o possible to con str uct a unary or binary expression wit h an invalid operator such as ! or < (although the Parse func tion mention ed

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

200

CHAPTER 7. INTERFACES

below will never do this). Thes e er ror s caus e Eval to panic. Other erro rs, like evaluating a Var not present in the env iro nment, merely cause Eval to retur n the wro ng result. All of these er ror s could be detec ted by ins pec ting the Expr before evaluating it. That will be the job of the Check method, which we will show soon, but firs t let’s test Eval.

The TestEval func tion below is a test of the evaluator. It uses the testing package, which we’ll explain in Chapt er 11, but for now it’s enoug h to know that cal ling t.Errorf reports an er ror. The function loops over a table of inp uts that defines three expressions and dif ferent enviro nments for each one. The firs t expression computes the radius of a circle given its area A, the secon d comp utes the sum of the cub es of two var iables x and y, and the third converts a Fahren heit temperature F to Cel siu s.

func TestEval(t *testing.T) {

tests := []struct {

expr string

env Env

want string

}{

{"sqrt(A / pi)", Env{"A": 87616, "pi": math.Pi}, "167"},

{"pow(x, 3) + pow(y, 3)", Env{"x": 12, "y": 1}, "1729"},

{"pow(x, 3) + pow(y, 3)", Env{"x": 9, "y": 10}, "1729"},

{"5 / 9 * (F - 32)", Env{"F": -40}, "-40"},

{"5 / 9 * (F - 32)", Env{"F": 32}, "0"}, ptg16091132

{"5 / 9 * (F - 32)", Env{"F": 212}, "100"},

}

var prevExpr string

for _, test := range tests {

// Print expr only when it changes.

if test.expr != prevExpr {

fmt.Printf("\n%s\n", test.expr)

prevExpr = test.expr

}

expr, err := Parse(test.expr)

if err != nil {

t.Error(err) // parse error

continue

}

got := fmt.Sprintf("%.6g", expr.Eval(test.env)) fmt.Printf("\t%v => %s\n", test.env, got)

if got != test.want {

t.Errorf("%s.Eval() in %s = %q, want %q\n",

test.expr, test.env, got, test.want)

}

}

}

Fo r each ent ry in the table, the test parses the expression, evaluates it in the env iro nment, and pr ints the result. We don’t have space to show the Parse func tion here, but you’ll find it if you down load the package using go get.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.9. EXAMPLE: EXPRESSION EVALUATOR

201

The go test command (§11.1) runs a package’s tests:

$ go test -v gopl.io/ch7/eval

The -v flag lets us see the print ed out put of the test, which is nor mal ly sup pressed for a successf ul test like this one. Here is the out put of the test’s fmt.Printf st atements: sqrt(A / pi)

map[A:87616 pi:3.141592653589793] => 167

pow(x, 3) + pow(y, 3)

map[x:12 y:1] => 1729

map[x:9 y:10] => 1729

5 / 9 * (F - 32)

map[F:-40] => -40

map[F:32] => 0

map[F:212] => 100

Fo rtunate ly the inp uts so far have all been wel l formed, but our luck is unlikely to last. Even in interprete d languages, it is common to che ck the syntax for stati c er ror s, that is, mist akes that can be detec ted wit hout running the program. By sep arat ing the static che cks fro m the dy namic ones, we can detec t er ror s so oner and per for m many che cks only once ins tead of ptg16091132

each time an expression is evaluate d.

Let’s add anot her met hod to the Expr interface. The Check method che cks for static error s in an expression syntax tre e. We’ll explain its vars parameter in a mom ent.

type Expr interface {

Eval(env Env) float64

// Check reports errors in this Expr and adds its Vars to the set.

Check(vars map[Var]bool) error

}

The con crete Check methods are shown below. Evaluation of literal and Var cannot fai l, so the Check methods for these typ es return nil. The methods for unary and binary firs t ch eck that the operator is valid, then rec ursively che ck the operands. Simi larly, the method for call firs t ch ecks that the function is known and has the rig ht number of arguments, then rec ursive ly che cks each argument.

func (v Var) Check(vars map[Var]bool) error {

vars[v] = true

return nil

}

func (literal) Check(vars map[Var]bool) error {

return nil

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

202

CHAPTER 7. INTERFACES

func (u unary) Check(vars map[Var]bool) error {

if !strings.ContainsRune("+-", u.op) {

return fmt.Errorf("unexpected unary op %q", u.op)

}

return u.x.Check(vars)

}

func (b binary) Check(vars map[Var]bool) error {

if !strings.ContainsRune("+-*/", b.op) {

return fmt.Errorf("unexpected binary op %q", b.op)

}

if err := b.x.Check(vars); err != nil {

return err

}

return b.y.Check(vars)

}

func (c call) Check(vars map[Var]bool) error {

arity, ok := numParams[c.fn]

if !ok {

return fmt.Errorf("unknown function %q", c.fn)

}

if len(c.args) != arity {

return fmt.Errorf("call to %s has %d args, want %d", c.fn, len(c.args), arity)

ptg16091132

}

for _, arg := range c.args {

if err := arg.Check(vars); err != nil {

return err

}

}

return nil

}

var numParams = map[string]int{"pow": 2, "sin": 1, "sqrt": 1}

We’ve liste d a selec tion of flawed inp uts and the error s they elicit, in two gro ups. The Parse func tion (not shown) rep orts syntax error s and the Check func tion rep orts semantic error s.

x % 2

unexpected '%'

math.Pi

unexpected '.'

!true

unexpected '!'

"hello"

unexpected '"'

log(10)

unknown function "log"

sqrt(1, 2)

call to sqrt has 2 args, want 1

Check’s argument, a set of Vars, accumulates the set of var iable names found wit hin the expression. Each of these var iables must be present in the env iro nment for evaluation to succe e d. This set is log ical ly the re sult of the cal l to Check, but because the method is rec ursive, it is more convenient for Check to popu late a set passed as a parameter. The client must provide an empt y set in the initial cal l.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.9. EXAMPLE: EXPRESSION EVALUATOR

203

In Sec tion 3.2, we plott ed a function f(x,y) that was fixe d at comp ile time. Now that we can pars e, ch eck, and evaluate expressions in str ings, we can bui ld a web app lic ation that receives an expression at run time fro m the client and plots the sur face of that function. We can use the vars set to che ck that the expression is a function of only two var iables, x and y—thre e, ac tually, since we’ll provide r, the radius, as a convenience. And we’ll use the Check method to rejec t ill-for med expressions before evaluation beg ins so that we don’t rep eat those che cks during the 40,000 evaluation s (100&100 cells, each wit h four corners) of the function that fol low.

The parseAndCheck func tion com bines these parsing and che cking steps: gopl.io/ch7/surface

import "gopl.io/ch7/eval"

func parseAndCheck(s string) (eval.Expr, error) {

if s == "" {

return nil, fmt.Errorf("empty expression")

}

expr, err := eval.Parse(s)

if err != nil {

return nil, err

}

vars := make(map[eval.Var]bool)

if err := expr.Check(vars); err != nil {

ptg16091132

return nil, err

}

for v := range vars {

if v != "x" && v != "y" && v != "r" {

return nil, fmt.Errorf("undefined variable: %s", v)

}

}

return expr, nil

}

To make this a web app lic ation, all we need is the plot func tion below, which has the fami liar sig nature of an http.HandlerFunc:

func plot(w http.ResponseWriter, r *http.Request) {

r.ParseForm()

expr, err := parseAndCheck(r.Form.Get("expr")) if err != nil {

http.Error(w, "bad expr: "+err.Error(), http.StatusBadRequest) return

}

w.Header().Set("Content-Type", "image/svg+xml") surface(w, func(x, y float64) float64 {

r := math.Hypot(x, y) // distance from (0,0)

return expr.Eval(eval.Env{"x": x, "y": y, "r": r})

})

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

204

CHAPTER 7. INTERFACES

ptg16091132

Figure 7.7. The sur faces of three functions: (a) sin(-x)*pow(1.5,-r); (b) pow(2,sin(y))*pow(2,sin(x))/12; (c) sin(x*y/10)/10.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.10. TYPE ASSERTIONS

205

The plot func tion parses and che cks the expression specified in the HTTP request and uses it to cre ate an anony mou s func tion of two var iables. The anony mou s func tion has the same sig-nature as the fixe d func tion f from the original sur face-plott ing program, but it evaluates the us er-supp lie d expression. The env iro nment defines x, y, and the radius r. Final ly, plot calls surface, which is just the main func tion fro m gopl.io/ch3/surface, modified to take the func tion to plot and the out put io.Writer as parameters, instead of using the fixe d func tion f and os.Stdout. Figure 7.7 shows three sur faces produce d by the program.

Exercis e 7.13: Add a String method to Expr to prett y-print the syntax tre e. Che ck that the resu lts, when parsed again, yield an equivalent tre e.

Exercis e 7.14: Define a new con crete typ e that sat isfies the Expr interface and provides a new op erat ion such as computing the minimum value of its operands. Since the Parse func tion do es not cre ate ins tances of this new typ e, to use it you will need to con str uct a syntax tre e direc tly (or extend the parser).

Exercis e 7.15: Wr ite a program that reads a single expression fro m the stand ard inp ut, prompts the user to provide values for any var iables, then evaluates the expression in the resu lting env iro nment. Handle all error s gracef ully.

Exercis e 7.16: Wr ite a web-b ased calc ulator program.

ptg16091132

7.10. Type Assertions

A ty pe ass ertion is an operat ion app lie d to an int erface value. Syntactic ally, it looks like x.(T), where x is an expression of an int erface typ e and T is a typ e, cal le d the ‘‘asserted’’ type. A type assertion che cks that the dynamic typ e of its operand match es the ass erted typ e.

There are two possibi lit ies. First, if the ass erted typ e T is a con crete typ e, then the typ e assertion che cks whether x’s dynamic typ e is ident ical to T. If this che ck succe e ds, the result of the type ass ertion is x’s dynamic value, whose typ e is of cours e T. In other words, a typ e assertion to a con crete typ e ext racts the con crete value fro m its operand. If the che ck fai ls, then the op erat ion panics. For example:

var w io.Writer

w = os.Stdout

f := w.(*os.File)

// success: f == os.Stdout

c := w.(*bytes.Buffer) // panic: interface holds *os.File, not *bytes.Buffer Second, if ins tead the ass erted typ e T is an int erface typ e, then the typ e assertion che cks whet her x’s dynamic typ e sati sfies T. If this che ck succe e ds, the dynamic value is not ext racte d; the result is still an int erface value wit h the same typ e and value comp onents, but the result has the int erface typ e T. In other words, a typ e assertion to an int erface typ e ch anges the typ e of the expression, mak ing a dif ferent (and usu ally larger) set of met hods accessible, but it pres erves the dynamic typ e and value comp onents inside the int erface value.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

206

CHAPTER 7. INTERFACES

Af ter the firs t type ass ertion below, bot h w and rw hold os.Stdout so each has a dynamic typ e of *os.File, but w, an io.Writer, exp oses only the file’s Write method, where as rw exp oses its Read method too.

var w io.Writer

w = os.Stdout

rw := w.(io.ReadWriter) // success: *os.File has both Read and Write w = new(ByteCounter)

rw = w.(io.ReadWriter) // panic: *ByteCounter has no Read method No matter what typ e was asserted, if the operand is a nil int erface value, the typ e assertion fai ls. A type ass ertion to a less restr ictive int erface typ e (one wit h fe wer met hods) is rarely ne e ded, as it beh aves just like an assig nment, except in the nil cas e.

w = rw

// io.ReadWriter is assignable to io.Writer

w = rw.(io.Writer) // fails only if rw == nil

Of ten we’re not sure of the dynamic typ e of an int erface value, and we’d like to test whether it is som e partic ular typ e. If the typ e assertion app ears in an assig nment in which two results are exp ected, such as the fol low ing declarat ions, the operat ion does not panic on fai lure but instead retur ns an addition al secon d resu lt, a boole an indic ating success: var w io.Writer = os.Stdout

f, ok := w.(*os.File)

// success:

ok, f == os.Stdout

b, ok := w.(*bytes.Buffer) // failure: !ok, b == nil

ptg16091132

The secon d resu lt is convent ion ally assig ned to a var iable named ok. If the operat ion fai le d, ok is false, and the firs t resu lt is equ al to the zero value of the ass erted typ e, which in this example is a nil *bytes.Buffer.

The ok resu lt is often immediate ly used to decide what to do next. The extende d form of the if st atement makes this quite compact:

if f, ok := w.(*os.File); ok {

// ...use f...

}

Wh en the operand of a typ e assertion is a var iable, rat her than inv ent another name for the ne w lo cal variable, you’ll som etimes see the original name reused, shadowing the original, like this:

if w, ok := w.(*os.File); ok {

// ...use w...

}

7.11. Discriminating Errors with Type Assertions

Consider the set of error s returned by file operat ions in the os package. I/O can fail for any numb er of reasons, but three kinds of fai lure often must be handled dif ferently: file already exists (for cre ate operat ions), file not found (for read operat ions), and per mission denie d. The

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.11. DISCRIMINATING ERRORS WITH TYPE ASSERTIONS

207

os package provides these three helper functions to classif y the fai lure indic ated by a given error value:

package os

func IsExist(err error) bool

func IsNotExist(err error) bool

func IsPermission(err error) bool

A naïve imp lementation of one of these pre dic ates might che ck that the error message contains a cer tain subst ring,

func IsNotExist(err error) bool {

// NOTE: not robust!

return strings.Contains(err.Error(), "file does not exist")

}

but because the log ic for handling I/O erro rs can vary fro m on e pl atfor m to another, this approach is not robust and the same fai lure may be rep orted wit h a var iety of dif ferent error mess ages. Che cking for subst rings of error messages may be useful dur ing testing to ens ure that functions fai l in the exp ected manner, but it’s inade quate for produc tion code.

A more reliable appro ach is to represent str uctured error values using a dedic ated typ e. The os package defines a typ e called PathError to des crib e fai lures invo l ving an operat ion on a ptg16091132

file pat h, li ke Open or Delete, and a var iant cal le d LinkError to des crib e fai lures of operation s invo l ving two file pat hs, like Symlink and Rename. Here’s os.PathError: package os

// PathError records an error and the operation and file path that caused it.

type PathError struct {

Op

string

Path string

Err error

}

func (e *PathError) Error() string {

return e.Op + " " + e.Path + ": " + e.Err.Error()

}

Most clients are obliv iou s to PathError and deal wit h al l er ror s in a unifor m way by cal ling their Error methods. Alt hough PathError’s Error method for ms a message by simply concatenating the fields, PathError’s str ucture preserves the underly ing components of the error.

Clients that need to distinguish one kind of fai lure fro m anot her can use a typ e assertion to detec t the specific typ e of the error ; the specific typ e prov ides more det ail than a simple str ing .

_, err := os.Open("/no/such/file")

fmt.Println(err) // "open /no/such/file: No such file or directory"

fmt.Printf("%#v\n", err)

// Output:

// &os.PathError{Op:"open", Path:"/no/such/file", Err:0x2}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

208

CHAPTER 7. INTERFACES

That’s how the three helper functions work. For example, IsNotExist, shown below, rep orts whet her an erro r is equ al to syscall.ENOENT (§7.8) or to the distinguishe d er ror os.ErrNotExist (s ee io.EOF in §5.4.2), or is a *PathError whos e underly ing error is one of thos e two.

import (

"errors"

"syscall"

)

var ErrNotExist = errors.New("file does not exist")

// IsNotExist returns a boolean indicating whether the error is known to

// report that a file or directory does not exist. It is satisfied by

// ErrNotExist as well as some syscall errors.

func IsNotExist(err error) bool {

if pe, ok := err.(*PathError); ok {

err = pe.Err

}

return err == syscall.ENOENT || err == ErrNotExist

}

And here it is in action:

_, err := os.Open("/no/such/file")

fmt.Println(os.IsNotExist(err)) // "true"

ptg16091132

Of course, PathError’s str ucture is lost if the error message is com bine d into a larger str ing , for ins tance by a cal l to fmt.Errorf. Error dis criminat ion must usu ally be don e immediate ly af ter the fai ling operat ion, before an error is pro pagated to the cal ler.

7.12. Querying Behaviors with Int erface Type Assertions The log ic below is simi lar to the par t of the net/http we b server responsible for writing HT TP he ader fields such as "Content-type: text/html". The io.Writer w repres ents the HT TP resp ons e; the bytes writt en to it are ult imate ly sent to som eon e’s web brows er.

func writeHeader(w io.Writer, contentType string) error {

if _, err := w.Write([]byte("Content-Type: ")); err != nil {

return err

}

if _, err := w.Write([]byte(contentType)); err != nil {

return err

}

// ...

}

Becaus e the Write method requires a byte slice, and the value we wish to write is a str ing , a

[]byte(...) conv ersion is required. This conversion allocates memor y and makes a copy, but the copy is throw n away almost immediate ly after. Let’s pretend that this is a core par t of

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.12. QUERYING BEHAVIORS WITH INTERFACE TYPE ASSERTIONS

209

the web ser ver and that our profiling has reveale d that this memor y al location is slowing it down. Can we avoid allo cat ing memor y here?

The io.Writer interface tel ls us only one fac t ab out the con crete typ e that w holds: that bytes may be writt en to it. If we look behind the cur tains of the net/http package, we see that the dy namic typ e that w holds in this program also has a WriteString method that allows str ings to be efficiently writt en to it, avoiding the need to allocate a temporar y copy. (This may seem li ke a shot in the dark, but a number of imp ortant typ es that sat isf y io.Writer also have a WriteString method, including *bytes.Buffer, *os.File and *bufio.Writer.) We cannot assume that an arbit rar y io.Writer w also has the WriteString method. But we can define a new int erface that has just this met hod and use a typ e assertion to test whether the dynamic typ e of w satisfies this new int erface.

// writeString writes s to w.

// If w has a WriteString method, it is invoked instead of w.Write.

func writeString(w io.Writer, s string) (n int, err error) {

type stringWriter interface {

WriteString(string) (n int, err error)

}

if sw, ok := w.(stringWriter); ok {

return sw.WriteString(s) // avoid a copy

ptg16091132

}

return w.Write([]byte(s)) // allocate temporary copy

}

func writeHeader(w io.Writer, contentType string) error {

if _, err := writeString(w, "Content-Type: "); err != nil {

return err

}

if _, err := writeString(w, contentType); err != nil {

return err

}

// ...

}

To avoid rep eat ing ourselves, we’ve mov ed the che ck int o the utilit y func tion writeString, but it is so useful that the stand ard librar y prov ides it as io.WriteString. It is the recommended way to write a str ing to an io.Writer.

Wh at’s cur ious in this example is that there is no stand ard int erface that defines the WriteString method and specifies its required beh avior. Fur thermore, whether or not a concrete typ e satisfies the stringWriter interface is deter mined only by its met hods, not by any de clare d re lat ions hip bet ween it and the int erface typ e. What this means is that the technique ab ove relies on the assump tion that if a typ e satisfies the int erface below, th en WriteString(s) mu st have the same effec t as Write([]byte(s)).

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

210

CHAPTER 7. INTERFACES

interface {

io.Writer

WriteString(s string) (n int, err error)

}

Although io.WriteString do cuments its assump tion, few functions that cal l it are likely to do cument that the y to o make the same assump tion. Defining a met hod of a par tic ular typ e is taken as an imp licit ass ent for a cer tain beh avioral cont rac t. Ne wcomers to Go, esp eci ally thos e from a backg round in stron gly typ ed langu ages, may find this lack of explicit int ent ion unsett ling, but it is rarely a problem in prac tice. Wit h the exception of the emp ty int erface interface{}, int erface typ es are seldom sat isfied by unintended coincidence.

The writeString func tion above uses a typ e assertion to see whether a value of a general interface typ e also sat isfies a more specific int erface typ e, and if so, it uses the beh avior s of the sp ecific int erface. This technique can be put to good use whether or not the quer ied int erface is stand ard like io.ReadWriter or user-define d li ke stringWriter.

It’s als o how fmt.Fprintf dist inguishes values that sat isf y error or fmt.Stringer from all ot her values. Wit hin fmt.Fprintf, there is a step that converts a single operand to a str ing , something like this:

package fmt

func formatOneValue(x interface{}) string {

ptg16091132

if err, ok := x.(error); ok {

return err.Error()

}

if str, ok := x.(Stringer); ok {

return str.String()

}

// ...all other types...

}

If x satisfies either of the two int erfaces, that deter mines the for matting of the value. If not, the defau lt cas e hand les all other typ es more or less unifor mly using reflec tion; we’ll find out how in Chapt er 12.

Again, this makes the assump tion that any typ e with a String method sat isfies the beh avioral cont rac t of fmt.Stringer, which is to retur n a str ing suitable for printing .

7.13. Type Switches

Interfaces are used in two distinc t st yles. In the firs t st yle, exemp lified by io.Reader, io.Writer, fmt.Stringer, sort.Interface, http.Handler, and error, an int erface’s met hods express the simi lar ities of the con crete typ es that sat isf y the int erface but hide the represent ation det ails and int rinsic operat ions of those con crete typ es. The emp hasis is on the methods, not on the con crete typ es.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.13. TYPE SWITCHES

211

The secon d st yle exploits the abi lit y of an int erface value to hold values of a var iety of con crete types and con siders the int erface to be the union of those typ es. Type ass ertions are used to discriminate among these typ es dy namic ally and tre at each cas e dif ferently. In this sty le, the emph asis is on the con crete typ es that sat isf y the int erface, not on the int erface’s met hods (if inde e d it has any), and there is no hiding of infor mat ion. We’ll des crib e interfaces used this way as di scr imi nat ed uni ons.

If you’re fami liar wit h objec t-oriente d prog ramming, you may recog nize thes e two sty les as subtype polymor phism and ad hoc polymor phism, but you needn’t remember those ter ms. For the remainder of this chapt er, we’ll present examples of the secon d st yle.

Go’s API for quer ying an SQL dat abas e, like those of other langu ages, lets us cle anly sep arate the fixe d part of a quer y from the var iable par ts. An example client mig ht look like this: import "database/sql"

func listTracks(db sql.DB, artist string, minYear, maxYear int) {

result, err := db.Exec(

"SELECT * FROM tracks WHERE artist = ? AND ? <= year AND year <= ?", artist, minYear, maxYear)

// ...

}

The Exec method replaces each '?' in the quer y st ring wit h an SQL lit eral denot ing the cor-ptg16091132

resp onding argument value, which may be a boole an, a number, a str ing , or nil. Con str ucting quer ies this way helps avoid SQL inj e ction att acks, in which an adversary takes cont rol of the quer y by exploit ing imp rop er quotation of inp ut dat a. Wi thin Exec, we mig ht find a function like the one below, which converts each argument value to its literal SQL not ation.

func sqlQuote(x interface{}) string {

if x == nil {

return "NULL"

} else if _, ok := x.(int); ok {

return fmt.Sprintf("%d", x)

} else if _, ok := x.(uint); ok {

return fmt.Sprintf("%d", x)

} else if b, ok := x.(bool); ok {

if b {

return "TRUE"

}

return "FALSE"

} else if s, ok := x.(string); ok {

return sqlQuoteString(s) // (not shown)

} else {

panic(fmt.Sprintf("unexpected type %T: %v", x, x))

}

}

A switch st atement simplifies an if-else ch ain that per for ms a ser ies of value equ ality tests.

An analogou s ty pe switch st atement simplifies an if-else ch ain of typ e assertions.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

212

CHAPTER 7. INTERFACES

In its simplest for m, a typ e sw itch looks like an ordinar y sw itch statement in which the operand is x.(type)—that’s lit eral ly the key word type—and each cas e has one or more typ es. A type switch enables a multi-way branch bas ed on the int erface value’s dynamic typ e. The nil case match es if x == nil, and the default case match es if no other cas e do es. A typ e sw itch for sqlQuote would have these cas es:

switch x.(type) {

case nil:

// ...

case int, uint: // ...

case bool:

// ...

case string:

// ...

default:

// ...

}

As wit h an ordinar y sw itch statement (§1.8), cas es are con sidered in order and, when a match is found, the cas e’s body is exe cut ed. Cas e order becom es sig nificant when one or more cas e types are int erfaces, since then there is a possibi lit y of two cas es matching. The position of the default case rel ative to the others is immater ial. No fallthrough is allow ed.

No tice that in the original function, the log ic for the bool and string cases needs access to the value extrac ted by the typ e assertion. Since this is typic al, the typ e sw itch statement has an extende d form that binds the ext racte d value to a new var iable wit hin each cas e: ptg16091132

switch x := x.(type) { /* ... */ }

Here we’ve cal le d the new var iables x to o; as wit h type ass ertions, reuse of var iable names is common. Like a switch st atement, a typ e sw itch imp licitly cre ates a lexic al block, so the declarat ion of the new var iable cal le d x do es not conflic t with a var iable x in an out er blo ck. Each case also imp licitly cre ates a sep arate lexic al block.

Re writ ing sqlQuote to use the extende d form of typ e sw itch makes it sig nificant ly cle arer : func sqlQuote(x interface{}) string {

switch x := x.(type) {

case nil:

return "NULL"

case int, uint:

return fmt.Sprintf("%d", x) // x has type interface{} here.

case bool:

if x {

return "TRUE"

}

return "FALSE"

case string:

return sqlQuoteString(x) // (not shown)

default:

panic(fmt.Sprintf("unexpected type %T: %v", x, x))

}

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.14. EXAMPLE: TOKEN-BASED XML DECODING

213

In this version, wit hin the blo ck of each single-t ype cas e, the var iable x has the same typ e as the cas e. For ins tance, x has typ e bool within the bool case and string within the string case. In all other cas es, x has the (interface) typ e of the switch op erand, which is interface{} in this example. When the same action is required for multiple cas es, li ke int and uint, the typ e sw itch makes it easy to com bine them.

Although sqlQuote accepts an argument of any typ e, the function runs to complet ion only if the argument’s typ e match es one of the cas es in the typ e sw itch; other wis e it panics wit h an

‘‘unexp ected typ e’’ mess age . Although the typ e of x is interface{}, we con sider it a di scr imi nat ed uni on of int, uint, bool, string, and nil.

7.14. Example: Token-Based XML Decoding

Section 4.5 showe d how to decode JSON documents into Go dat a st ruc tures wit h the Marshal and Unmarshal func tions fro m the encoding/json package. The encoding/xml package prov ides a similar API. This appro ach is convenient when we want to con str uct a representation of the document tre e, but that’s unnecessary for many programs. The encoding/xml package als o prov ides a lower-le vel tok en-bas ed API for decoding XML. In the token-b ased st yle, the parser con sumes the inp ut and pro duces a stream of tokens, primarily of four kind s—StartElement, EndElement, CharData, and Comment—each being a con crete typ e in ptg16091132

the encoding/xml package. Each cal l to (*xml.Decoder).Token returns a token.

The relevant par ts of the API are shown here:

encoding/xml

package xml

type Name struct {

Local string // e.g., "Title" or "id"

}

type Attr struct { // e.g., name="value"

Name Name

Value string

}

// A Token includes StartElement, EndElement, CharData,

// and Comment, plus a few esoteric types (not shown).

type Token interface{}

type StartElement struct { // e.g., <name>

Name Name

Attr []Attr

}

type EndElement struct { Name Name } // e.g., </name> type CharData []byte

// e.g., <p>CharData</p>

type Comment []byte

// e.g., <!-- Comment -->

type Decoder struct{ /* ... */ }

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

214

CHAPTER 7. INTERFACES

func NewDecoder(io.Reader) *Decoder

func (*Decoder) Token() (Token, error) // returns next Token in sequence The Token interface, which has no met hods, is als o an example of a dis criminated union. The purpos e of a tradition al int erface like io.Reader is to hide detai ls of the con crete typ es that satisf y it so that new imp lementation s can be cre ate d; each con crete typ e is tre ate d unifor mly.

By contrast, the set of con crete typ es that sat isf y a dis criminated union is fixe d by the desig n and exp osed, not hidden. Discriminated union typ es have few met hods; functions that operate on them are express ed as a set of cas es using a typ e sw itch, wit h dif ferent log ic in each cas e.

The xmlselect prog ram below ext racts and prints the text found beneath cer tain elements in an XML document tre e. Using the API above , it can do its job in a single pass over the inp ut without ever mater ializing the tre e.

gopl.io/ch7/xmlselect

// Xmlselect prints the text of selected elements of an XML document.

package main

import (

"encoding/xml"

"fmt"

"io"

"os"

ptg16091132

"strings"

)

func main() {

dec := xml.NewDecoder(os.Stdin)

var stack []string // stack of element names

for {

tok, err := dec.Token()

if err == io.EOF {

break

} else if err != nil {

fmt.Fprintf(os.Stderr, "xmlselect: %v\n", err) os.Exit(1)

}

switch tok := tok.(type) {

case xml.StartElement:

stack = append(stack, tok.Name.Local) // push

case xml.EndElement:

stack = stack[:len(stack)-1] // pop

case xml.CharData:

if containsAll(stack, os.Args[1:]) {

fmt.Printf("%s: %s\n", strings.Join(stack, " "), tok)

}

}

}

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 7.14. EXAMPLE: TOKEN-BASED XML DECODING

215

// containsAll reports whether x contains the elements of y, in order.

func containsAll(x, y []string) bool {

for len(y) <= len(x) {

if len(y) == 0 {

return true

}

if x[0] == y[0] {

y = y[1:]

}

x = x[1:]

}

return false

}

Each time the loop in main encounters a StartElement, it pushes the element’s name ont o a st ack, and for each EndElement it pops the name fro m the stack. The API guarante es that the sequence of StartElement and EndElement tokens will be pro perly match ed, even in ill-formed documents. Comments are ignored. When xmlselect encounters a CharData, it pr ints the text only if the stack contains all the elements named by the command-line arguments, in order.

The command below prints the text of any h2 elements appear ing beneath two level s of div elements. Its input is the XML specification, its elf an XML document.

ptg16091132

$ go build gopl.io/ch1/fetch

$ ./fetch http://www.w3.org/TR/2006/REC-xml11-20060816 |

./xmlselect div div h2

html body div div h2: 1 Introduction

html body div div h2: 2 Documents

html body div div h2: 3 Logical Structures

html body div div h2: 4 Physical Structures

html body div div h2: 5 Conformance

html body div div h2: 6 Notation

html body div div h2: A References

html body div div h2: B Definitions for Character Normalization

...

Exercis e 7.17: Extend xmlselect so that elements may be selec ted not just by name, but by their attr ibutes too, in the manner of CSS, so that, for ins tance, an element like

<div id="page" class="wide"> could be selec ted by a matching id or class as wel l as its name.

Exercis e 7.18: Using the token-b ased decoder API, write a program that will read an arbit rar y XML document and con str uct a tre e of gener ic no des that represents it. Nodes are of two kind s: CharData no des represent text str ings, and Element no des represent named elements and their attr ibutes. Each element node has a slice of chi ld no des.

Yo u may find the fol low ing declarat ions helpf ul.

import "encoding/xml"

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

216

CHAPTER 7. INTERFACES

type Node interface{} // CharData or *Element

type CharData string

type Element struct {

Type

xml.Name

Attr

[]xml.Attr

Children []Node

}

7.15. A Fe w Words of Advice

Wh en desig ning a new package, nov ice Go programmers often start by cre ating a set of int erfaces and only later define the con crete typ es that sat isf y them. This appro ach results in many interfaces, each of which has only a single imp lementation. Don’t do that. Such int erfaces are unnecessary abstrac tions; the y also have a run-t ime cost. You can restr ict which met hods of a type or fields of a str uct are visible outside a package using the exp ort mechanism (§6.6).

Interfaces are only needed when there are two or more con crete typ es that must be dealt wit h in a unifor m way.

We make an exception to this rule when an int erface is sat isfied by a single con crete typ e but that typ e cannot live in the same package as the int erface because of its dependencies. In that ptg16091132

case, an int erface is a good way to decouple two packages.

Becaus e interfaces are used in Go only when the y are sat isfied by two or more typ es, they ne cessarily abstrac t away fro m the det ails of any par tic ular imp lementation. The result is smal ler interfaces wit h fe wer, simpler met hods, often just one as wit h io.Writer or fmt.Stringer. Small int erfaces are easier to sat isf y when new typ es come along . A good rule of thumb for int erface design is ask only for what you need.

This con cludes our tour of met hods and int erfaces. Go has gre at sup por t for the obj e ct-or iente d st yle of programming, but this does not mean you need to use it exc lusively. Not ever ything need be an obj e ct; stand alone functions have their place, as do unenc apsulated data typ es. Observe that toget her, the examples in the firs t five chapt ers of this book cal l no more than two dozen met hods, like input.Scan, as opp osed to ordinar y func tion cal ls like fmt.Printf.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

8

Goroutines and Channels

Conc urrent programming, the expression of a program as a comp osition of several autonomous activ ities, has never been more imp ortant than it is today. Web ser vers handle re quests for thous and s of clients at once. Tablet and phone apps render animat ions in the user interface whi le simultane ously per for ming computation and networ k re quests in the back-ptg16091132

ground. Even tradition al batch pro blems—re ad som e data, compute, write som e output—us e conc urrency to hide the latency of I/O operat ions and to exploit a moder n comp uter’s many processors, which every year grow in number but not in speed.

Go enables two sty les of con cur rent programming. This chapt er presents goroutines and ch annel s, which sup por t commu nicating sequent ial pro cesses or CSP, a model of con cur rency in which values are passed bet ween indep endent activ ities (goroutines) but var iables are for the most par t confine d to a single activ ity. Chapt er 9 cov ers som e aspects of the more tradition al model of sh are d memory multithre ading, which will be fami liar if you’ve used threads in ot her mainst ream langu ages. Chapt er 9 als o points out som e important hazards and pit fal ls of conc urrent programming that we won’t delve int o in this chapt er.

Ev en thoug h Go’s sup por t for con cur rency is one of its gre at strengt hs, reasoning about concurrent programs is inherently harder than about sequential ones, and int uit ion s acquired from sequential programming may at times lead us ast ray. If this is your firs t encounter wit h conc urrency, we recommend spending a lit tle extra time thin king about the examples in these two chapt ers.

8.1. Goroutines

In Go, each con cur rently exe cut ing activ ity is cal le d a goro utine. Con sider a program that has two functions, one that does som e comp utation and one that writes som e output, and assume that neither function cal ls the other. A sequential program may cal l on e func tion and then 217

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

218

CHAPTER 8. GOROUTINES AND CHANNELS

call the other, but in a concur rent prog ram with two or more goroutines, cal ls to both func tions can be active at the same time. We’ll see such a program in a mom ent.

If you have used operat ing system threads or threads in other langu ages, then you can assume for now that a goroutine is simi lar to a thread, and you’ll be able to write cor rec t prog rams.

The dif ferences bet ween threads and goroutines are ess ent ial ly quant itative , not qualitative , and will be des crib ed in Sec tion 9.8.

Wh en a program starts, its only goroutine is the one that cal ls the main func tion, so we cal l it the main goroutine. New goroutines are cre ate d by the go st atement. Syntactic ally, a go st atement is an ordinar y func tion or met hod cal l prefixe d by the key word go. A go st atement caus es the function to be cal le d in a newly cre ate d goro utine. The go st atement its elf completes immediate ly:

f()

// call f(); wait for it to return

go f() // create a new goroutine that calls f(); don't wait In the example below, the main goroutine computes the 45t h Fibonacci number. Since it uses the ter r ibly inefficient rec ursive algor it hm, it runs for an appre ciable time, dur ing which we’d li ke to provide the user wit h a visual indic ation that the program is still running, by displ aying an animated textual ‘‘spinner.’’

gopl.io/ch8/spinner

ptg16091132

func main() {

go spinner(100 * time.Millisecond)

const n = 45

fibN := fib(n) // slow

fmt.Printf("\rFibonacci(%d) = %d\n", n, fibN)

}

func spinner(delay time.Duration) {

for {

for _, r := rangè-\|/` {

fmt.Printf("\r%c", r)

time.Sleep(delay)

}

}

}

func fib(x int) int {

if x < 2 {

return x

}

return fib(x-1) + fib(x-2)

}

Af ter several secon ds of animat ion, the fib(45) call retur ns and the main func tion prints its resu lt:

Fibonacci(45) = 1134903170

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.2. EXAMPLE: CONCURRENT CLOCK SERVER

219

The main func tion then retur ns. When this happens, all goroutines are abr uptly ter minated and the program exits. Other than by retur ning fro m main or exiting the program, there is no prog rammat ic way for one goroutine to stop another, but as we will see later, there are ways to communic ate wit h a goroutine to request that it stop its elf.

No tice how the program is express ed as the composition of two aut onom ous activ ities, spinning and Fib onacci comp utation. Each is writt en as a sep arate function but bot h make prog ress con cur rently.

8.2. Example: Concurrent Clock Server

Networ king is a natural domain in which to use con cur rency since ser vers typic ally handle many conne ction s from their clients at once, each client being ess ent ial ly indep endent of the ot hers. In this sec tion, we’ll int roduce the net package, which provides the components for building net wor ked client and ser ver programs that communic ate over TCP, UDP, or Unix domain sockets. The net/http package we’ve been using since Chapt er 1 is bui lt on top of func tions fro m the net package.

Our firs t example is a sequential clo ck ser ver that writes the cur rent time to the client once per second:

ptg16091132

gopl.io/ch8/clock1

// Clock1 is a TCP server that periodically writes the time.

package main

import (

"io"

"log"

"net"

"time"

)

func main() {

listener, err := net.Listen("tcp", "localhost:8000") if err != nil {

log.Fatal(err)

}

for {

conn, err := listener.Accept()

if err != nil {

log.Print(err) // e.g., connection aborted

continue

}

handleConn(conn) // handle one connection at a time

}

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

220

CHAPTER 8. GOROUTINES AND CHANNELS

func handleConn(c net.Conn) {

defer c.Close()

for {

_, err := io.WriteString(c, time.Now().Format("15:04:05\n")) if err != nil {

return // e.g., client disconnected

}

time.Sleep(1 * time.Second)

}

}

The Listen func tion cre ates a net.Listener, an obj e ct that listens for incoming conne ction s on a net wor k port, in this cas e TCP por t localhost:8000. The listener’s Accept method blocks until an incoming conne ction request is made, then retur ns a net.Conn objec t represent ing the conne ction.

The handleConn func tion handles one complete client conne ction. In a loop, it writes the current time, time.Now(), to the client. Since net.Conn satisfies the io.Writer interface, we can wr ite direc tly to it. The loop end s when the write fai ls, most likely because the client has disconnec ted, at which point handleConn clos es its side of the conne ction using a defer red cal l to Close and goes back to waiting for another connec tion request.

The time.Time.Format method provides a way to for mat date and time infor mat ion by ptg16091132

example. Its argument is a templ ate indic ating how to for mat a reference time, specifically Mon Jan 2 03:04:05PM 2006 UTC-0700. The reference time has eight components (day of the we ek, month, day of the mont h, and so on). Any col lec tion of them can appear in the Format st ring in any order and in a number of for mats; the selec ted components of the date and time wi l l be displ aye d in the selec ted for mats. Here we are just using the hour, minut e , and secon d of the time. The time package defines templ ates for many stand ard time for mats, such as time.RFC1123. The same mechanism is used in reverse when parsing a time using time.Parse.

To conne ct to the ser ver, we’ll need a client program such as nc (‘‘netc at’’), a stand ard utilit y prog ram for manipu lat ing net wor k connec tion s:

$ go build gopl.io/ch8/clock1

$ ./clock1 &

$ nc localhost 8000

13:58:54

13:58:55

13:58:56

13:58:57

^C

The client displ ays the time sent by the ser ver each secon d until we int err upt the client wit h Cont rol-C, which on Unix systems is ech oed as ^C by the shell. If nc or netcat is not ins tal le d on your system, you can use telnet or this simple Go version of netcat that uses net.Dial to conne ct to a TCP ser ver :

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.2. EXAMPLE: CONCURRENT CLOCK SERVER

221

gopl.io/ch8/netcat1

// Netcat1 is a read-only TCP client.

package main

import (

"io"

"log"

"net"

"os"

)

func main() {

conn, err := net.Dial("tcp", "localhost:8000") if err != nil {

log.Fatal(err)

}

defer conn.Close()

mustCopy(os.Stdout, conn)

}

func mustCopy(dst io.Writer, src io.Reader) {

if _, err := io.Copy(dst, src); err != nil {

log.Fatal(err)

}

ptg16091132

}

This program reads dat a from the conne ction and writes it to the stand ard out put unt i l an end-of-file condit ion or an error occ urs. The mustCopy func tion is a utilit y us ed in several examples in this sec tion. Let’s run two clients at the same time on dif ferent ter minals, one shown to the lef t and one to the rig ht:

$ go build gopl.io/ch8/netcat1

$ ./netcat1

13:58:54

$ ./netcat1

13:58:55

13:58:56

^C

13:58:57

13:58:58

13:58:59

^C

$ killall clock1

The killall command is a Unix utilit y that kills all pro cesses wit h the given name.

The secon d client must wait unt i l the firs t client is finishe d becaus e the ser ver is sequ ent ial; it de als wit h on ly one client at a time. Just one small change is needed to make the ser ver concurrent: adding the go ke yword to the cal l to handleConn caus es each cal l to run in its own goro utine.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

222

CHAPTER 8. GOROUTINES AND CHANNELS

gopl.io/ch8/clock2

for {

conn, err := listener.Accept()

if err != nil {

log.Print(err) // e.g., connection aborted

continue

}

go handleConn(conn) // handle connections concurrently

}

No w, multiple clients can receive the time at once:

$ go build gopl.io/ch8/clock2

$ ./clock2 &

$ go build gopl.io/ch8/netcat1

$ ./netcat1

14:02:54

$ ./netcat1

14:02:55

14:02:55

14:02:56

14:02:56

14:02:57

^C

14:02:58

14:02:59

$ ./netcat1

14:03:00

14:03:00

14:03:01

14:03:01

ptg16091132

^C

14:03:02

^C

$ killall clock2

Exercis e 8.1: Mo dif y clock2 to accept a por t numb er, and write a program, clockwall, that ac ts as a client of several clo ck ser vers at once, reading the times fro m each one and displ aying the results in a table, akin to the wal l of clo cks seen in som e bu siness offices. If you have access to geographic ally distr ibute d comp uters, run inst ances remot ely ; ot her wis e run local inst ances on dif ferent por ts wit h fake time zon es.

$ TZ=US/Eastern

./clock2 -port 8010 &

$ TZ=Asia/Tokyo

./clock2 -port 8020 &

$ TZ=Europe/London ./clock2 -port 8030 &

$ clockwall NewYork=localhost:8010 London=localhost:8020 Tokyo=localhost:8030

Exercis e 8.2: Implement a con cur rent File Transfer Pro tocol (FTP) ser ver. The ser ver should interpret command s from each client such as cd to change direc tor y, ls to list a direc tor y, get to send the contents of a file, and close to clos e the conne ction. You can use the stand ard ftp command as the client, or write your own.

8.3. Example: Concurrent Echo Server

The clo ck ser ver used one goroutine per connec tion. In this sec tion, we’ll bui ld an ech o server that uses multiple goroutines per connec tion. Most ech o servers merely write whate ver the y

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.3. EXAMPLE: CONCURRENT ECHO SERVER

223

re ad, which can be don e with this trivial version of handleConn: func handleConn(c net.Conn) {

io.Copy(c, c) // NOTE: ignoring errors

c.Close()

}

A more int erest ing ech o server might simulate the reverb erations of a real ech o, wit h the resp ons e loud at firs t ("HELLO!"), then moderate ("Hello!") after a delay, then quiet ("hello!") before fading to not hing, as in this version of handleConn: gopl.io/ch8/reverb1

func echo(c net.Conn, shout string, delay time.Duration) {

fmt.Fprintln(c, "\t", strings.ToUpper(shout)) time.Sleep(delay)

fmt.Fprintln(c, "\t", shout)

time.Sleep(delay)

fmt.Fprintln(c, "\t", strings.ToLower(shout))

}

func handleConn(c net.Conn) {

input := bufio.NewScanner(c)

for input.Scan() {

echo(c, input.Text(), 1*time.Second)

}

ptg16091132

// NOTE: ignoring potential errors from input.Err()

c.Close()

}

We’ll need to upg rade our client program so that it sends ter minal input to the ser ver whi le also copying the ser ver respons e to the out put, which presents another opp ortunity to use conc urrency :

gopl.io/ch8/netcat2

func main() {

conn, err := net.Dial("tcp", "localhost:8000") if err != nil {

log.Fatal(err)

}

defer conn.Close()

go mustCopy(os.Stdout, conn)

mustCopy(conn, os.Stdin)

}

While the main goroutine reads the stand ard inp ut and sends it to the ser ver, a secon d goro utine reads and prints the ser ver’s respons e. When the main goroutine encount ers the end of the inp ut, for example, after the user typ es Cont rol-D (^D) at the ter minal (or the equivalent Control-Z on Micros oft Windows), the program stops, even if the other goroutine st i l l has wor k to do. (We’ll see how to make the program wait for bot h sides to finish once we’ve int roduce d ch annel s in Sec tion 8.4.1.)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

224

CHAPTER 8. GOROUTINES AND CHANNELS

In the session below, the client’s inp ut is lef t-aligne d and the ser ver’s respons es are indente d.

The client shouts at the ech o server three times:

$ go build gopl.io/ch8/reverb1

$ ./reverb1 &

$ go build gopl.io/ch8/netcat2

$ ./netcat2

Hello?

HELLO?

Hello?

hello?

Is there anybody there?

IS THERE ANYBODY THERE?

Yooo-hooo!

Is there anybody there?

is there anybody there?

YOOO-HOOO!

Yooo-hooo!

yooo-hooo!

^D

$ killall reverb1

No tice that the third shout fro m the client is not dealt wit h until the secon d shout has petered ptg16091132

out, which is not ver y re alist ic. A re al ech o would con sist of the comp ositi on of the three inde-pendent shouts. To simulate it, we’ll need more goroutines. Again, all we need to do is add the go ke yword , this time to the cal l to echo:

gopl.io/ch8/reverb2

func handleConn(c net.Conn) {

input := bufio.NewScanner(c)

for input.Scan() {

go echo(c, input.Text(), 1*time.Second)

}

// NOTE: ignoring potential errors from input.Err()

c.Close()

}

The arguments to the function started by go are evaluate d when the go st atement its elf is exe-cute d; thu s input.Text() is evaluate d in the main goroutine.

No w the ech oes are con cur rent and overl ap in time: $ go build gopl.io/ch8/reverb2

$ ./reverb2 &

$ ./netcat2

Is there anybody there?

IS THERE ANYBODY THERE?

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.4. CHANNELS

225

Yooo-hooo!

Is there anybody there?

YOOO-HOOO!

is there anybody there?

Yooo-hooo!

yooo-hooo!

^D

$ killall reverb2

Al l that was required to make the ser ver use con cur rency, not just to handle conne ction s from mu ltiple clients but even wit hin a single conne ction, was the ins ertion of two go ke yword s.

Ho wever in adding these key words, we had to con sider caref ully that it is safe to cal l methods of net.Conn conc urrently, which is not true for most typ es. We’ll dis cuss the cruci al concept of concur rency saf ety in the next chapt er.

8.4. Channels

If goroutines are the activ ities of a con cur rent Go program, ch ann els are the conne ction s between them. A ch annel is a communic ation mechanism that lets one goroutine send values to another goroutine. Each channel is a con duit for values of a par tic ular typ e, cal le d the ch annel’s el ement type. The typ e of a channel whose elements have typ e int is writt en ptg16091132

chan int.

To cre ate a channel, we use the bui lt-in make func tion: ch := make(chan int) // ch has type 'chan int'

As wit h maps, a channel is a reference to the dat a st ruc ture cre ate d by make. When we copy a ch annel or pass one as an argument to a function, we are copying a reference, so cal ler and callee refer to the same dat a st ruc ture. As wit h ot her reference typ es, the zero value of a channel is nil.

Tw o ch annel s of the same typ e may be compare d using ==. The comparison is true if bot h are references to the same channel dat a st ruc ture. A ch annel may also be compare d to nil.

A channel has two princip al operat ions, send and re ceive, col lec tive ly known as commu nicati ons. A send statement transmits a value fro m on e goro utine, through the channel, to another goroutine exe cut ing a cor responding receive expression. Bot h op erat ions are wr itt en using the <- op erator. In a send statement, the <- separates the channel and value operands. In a receive expression, <- precedes the channel operand. A re ceive expression whose resu lt is not used is a valid statement.

ch <- x

// a send statement

x = <-ch // a receive expression in an assignment statement

<-ch

// a receive statement; result is discarded

Channel s supp ort a third operat ion, cl ose, which sets a flag indic ating that no more values will ever be sent on this channel; subsequent att emp ts to send will panic. Receive operat ions on a

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

226

CHAPTER 8. GOROUTINES AND CHANNELS

clos ed channel yield the values that have been sent unt i l no more values are lef t; any receive op erat ions there after comp lete immediate ly and yield the zero value of the channel’s element type.

To clos e a channel, we cal l the bui lt-in close func tion: close(ch)

A channel cre ate d with a simple cal l to make is cal le d an unbuffered ch annel, but make accepts an opt ion al secon d argument, an integer cal le d the channel’s capacity. If the cap acity is non-zero, make creates a buffered ch annel.

ch = make(chan int)

// unbuffered channel

ch = make(chan int, 0) // unbuffered channel

ch = make(chan int, 3) // buffered channel with capacity 3

We’ll look at unbuf fered channel s firs t and buf fered channel s in Sec tion 8.4.4.

8.4.1. Unbuffered Channels

A send operat ion on an unbuf fered channel blo cks the sending goroutine unt i l anot her ptg16091132

goro utine exe cut es a corresponding receive on the same channel, at which point the value is transmitt ed and bot h goro utines may cont inue. Convers ely, if the receive operat ion was attemp ted firs t, the receiving goro utine is blo cke d until another goroutine per for ms a send on the same channel.

Communic ation over an unbuf fered channel causes the sending and receiving goro utines to sy nchro niz e. Because of this, unbuffered channel s are som etimes cal le d sy nchro nou s ch annel s.

Wh en a value is sent on an unbuf fered channel, the receipt of the value happ ens before the re awakening of the sending goroutine.

In dis cussions of con cur rency, when we say x hap pens before y, we don’t mean merely that x occurs earlier in time than y; we mean that it is guarante e d to do so and that all its prior ef fec ts, such as updates to var iables, are complete and that you may rely on them.

Wh en x neit her happens before y nor after y, we say that x is con cur rent wit h y. This doesn’t me an that x and y are necessarily simultane ous, merely that we cannot assume anything about their order ing. As we’ll see in the next chapt er, it’s necessary to order certain events dur ing the prog ram’s exe cut ion to avoid the pro blems that arise when two goro utines access the same var iable conc urrently.

The client program in Sec tion 8.3 copies input to the ser ver in its main goroutine, so the client prog ram terminates as soon as the inp ut stream clos es, even if the backg round goroutine is st i l l work ing . To make the program wait for the backg round goroutine to complete before exiting , we use a channel to synchro nize the two goro utines:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.4. CHANNELS

227

gopl.io/ch8/netcat3

func main() {

conn, err := net.Dial("tcp", "localhost:8000") if err != nil {

log.Fatal(err)

}

done := make(chan struct{})

go func() {

io.Copy(os.Stdout, conn) // NOTE: ignoring errors

log.Println("done")

done <- struct{}{} // signal the main goroutine

}()

mustCopy(conn, os.Stdin)

conn.Close()

<-done // wait for background goroutine to finish

}

Wh en the user clos es the stand ard inp ut stream, mustCopy returns and the main goroutine calls conn.Close(), closing bot h halves of the networ k connec tion. Closing the write half of the conne ction causes the ser ver to see an end-of-file condit ion. Closing the read half causes the backg round goroutine’s cal l to io.Copy to retur n a ‘‘re ad fro m clos ed conne ction’’ er ror, which is why we’ve remov ed the error log ging; Exercis e 8.3 sug gests a better solut ion. (No tice that the go st atement cal ls a lit eral function, a common con str uction.) ptg16091132

Before it retur ns, the backg round goroutine logs a message , then sends a value on the done ch annel. The main goroutine waits until it has receive d this value before retur ning. As a resu lt, the program always logs the "done" mess age before exiting .

Mess ages sent over channel s have two imp ortant asp ects. Each message has a value, but sometimes the fac t of communic ation and the moment at which it occ urs are just as important. We cal l mess ages events when we wish to stress this asp ect. When the event carries no addition al infor mat ion, that is, its sole pur pos e is synchro nizat ion, we’ll emp hasize this by using a channel whose element typ e is struct{}, thoug h it’s common to use a channel of bool or int for the same pur pos e since done <- 1 is shorter than done <- struct{}{}.

Exercis e 8.3: In netcat3, the int erface value conn has the con crete typ e *net.TCPConn, which repres ents a TCP connec tion. A TCP conne ction con sists of two halves that may be clos ed indep endently using its CloseRead and CloseWrite methods. Modif y the main goroutine of netcat3 to clos e on ly the write half of the conne ction so that the program will continue to pr int the final ech oes fro m the reverb1 server even after the stand ard inp ut has been clos ed.

(D oing this for the reverb2 server is harder ; see Exercis e 8.4.) 8.4.2. Pipelines

Channel s can be used to conne ct goroutines toget her so that the out put of one is the inp ut to anot her. This is cal le d a pip elin e. The program below con sists of three goroutines conne cte d by two channel s, as shown sch ematically in Figure 8.1.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

228

CHAPTER 8. GOROUTINES AND CHANNELS

Figure 8.1. A three-stage pip eline.

The firs t goro utine, counter, generates the int egers 0, 1, 2, ..., and sends them over a channel to the secon d goro utine, squarer, which receives each value, squ ares it, and sends the result over anot her channel to the third goroutine, pr int er, which receives the squ are d values and prints them. For clarity of this example, we have int ent ion ally chosen ver y simple functions, thoug h of cours e they are too computation ally trivial to war rant their own goroutines in a realist ic prog ram.

gopl.io/ch8/pipeline1

func main() {

naturals := make(chan int)

squares := make(chan int)

// Counter

go func() {

for x := 0; ; x++ {

ptg16091132

naturals <- x

}

}()

// Squarer

go func() {

for {

x := <-naturals

squares <- x * x

}

}()

// Printer (in main goroutine)

for {

fmt.Println(<-squares)

}

}

As you mig ht exp ect, the program prints the infinite ser ies of squ ares 0, 1, 4, 9, and so on.

Pipelines like this may be found in lon g-r unning ser ver programs where channel s are used for lifelon g communic ation bet ween goroutines containing infinite loops. But what if we want to send only a finite number of values through the pip eline?

If the sender knows that no fur ther values will ever be sent on a channel, it is useful to communic ate this fac t to the receiver goroutines so that the y can stop waiting . This is accom-plishe d by cl osing the channel using the bui lt-in close func tion:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.4. CHANNELS

229

close(naturals)

Af ter a channel has been clos ed, any fur ther send operat ions on it will panic. After the clos ed ch annel has been drai ned, that is, after the last sent element has been receive d, all subsequent re ceive operat ions will pro ceed wit hout blo cking but will yield a zero value. Closing the naturals ch annel above would cause the squ arer’s loop to spin as it receives a never-ending stream of zero values, and to send these zeros to the print er.

There is no way to test direc tly whether a channel has been clos ed, but there is a var iant of the re ceive operat ion that pro duces two results: the receive d ch annel element, plu s a boole an value, convent ion ally cal le d ok, which is true for a successf ul receive and false for a receive on a clos ed and drained channel. Using this feature , we can modif y the squ arer’s loop to stop when the naturals ch annel is drained and clos e the squares ch annel in tur n.

// Squarer

go func() {

for {

x, ok := <-naturals

if !ok {

break // channel was closed and drained

}

squares <- x * x

}

ptg16091132

close(squares)

}()

Becaus e the syntax above is clumsy and this pattern is common, the langu age lets us use a range lo op to iterate over channel s to o. This is a more convenient syntax for receiving all the values sent on a channel and ter minat ing the loop after the last one.

In the pip eline below, when the count er goroutine finishes its loop after 100 elements, it clos es the naturals ch annel, causing the squ arer to finish its loop and clos e the squares ch annel.

(In a more complex program, it mig ht make sense for the count er and squ arer functions to defer the cal ls to close at the outset.) Final ly, the main goroutine finishes its loop and the prog ram exits.

gopl.io/ch8/pipeline2

func main() {

naturals := make(chan int)

squares := make(chan int)

// Counter

go func() {

for x := 0; x < 100; x++ {

naturals <- x

}

close(naturals)

}()

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

230

CHAPTER 8. GOROUTINES AND CHANNELS

// Squarer

go func() {

for x := range naturals {

squares <- x * x

}

close(squares)

}()

// Printer (in main goroutine)

for x := range squares {

fmt.Println(x)

}

}

Yo u ne e dn’t clos e ever y ch annel when you’ve finishe d with it. It’s only necessary to clos e a ch annel when it is imp ortant to tel l the receiving goro utines that all dat a have been sent. A ch annel that the garb age col lec tor deter mines to be unreach able will have its res ources re claimed whether or not it is clos ed. (Don’t confuse this wit h the clos e op erat ion for open files. It is important to cal l the Close method on every file when you’ve finishe d with it.) At tempt ing to clos e an already-clos ed channel causes a panic, as does closing a nil channel.

Closing channel s has another use as a bro adc ast mechanism, which we’ll cov er in Sec tion 8.9.

ptg16091132

8.4.3. Unidirectional Channel Types

As programs grow, it is natural to bre ak up large functions int o smal ler pieces. Our pre vious example used three goroutines, communic ating over two channel s, which were local variables of main. The program natural ly div ides into three functions: func counter(out chan int)

func squarer(out, in chan int)

func printer(in chan int)

The squarer func tion, sitt ing in the midd le of the pip eline, takes two parameters, the inp ut ch annel and the out put channel. Bot h have the same typ e, but their intended uses are opp osite: in is only to be receive d from, and out is only to be sent to. The names in and out conv ey this int ent ion, but still, not hing pre vents squarer from sending to in or receiving from out.

This arrangement is typic al. When a channel is sup plie d as a function parameter, it is nearly always wit h the int ent that it be used exc lusively for sending or exc lusively for receiving .

To document this int ent and pre vent misus e, the Go typ e system provides unidirec tional ch annel typ es that expos e on ly one or the other of the send and receive operat ions. The typ e chan<- int, a send-only ch annel of int, allows sends but not receives. Convers ely, the typ e

<-chan int, a re ceive-only ch annel of int, allows receives but not sends. (The position of the

<- ar row rel ative to the chan ke yword is a mnemonic.) Violation s of this dis cip line are detec ted at compi le time.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.4. CHANNELS

231

Since the close op erat ion ass erts that no more sends will occ ur on a channel, only the sending goroutine is in a position to cal l it, and for this reason it is a compi le-t ime error to att emp t to clos e a receive-only channel.

Here’s the squ aring pip eline once more, this time wit h unidirec tion al channel typ es: gopl.io/ch8/pipeline3

func counter(out chan<- int) {

for x := 0; x < 100; x++ {

out <- x

}

close(out)

}

func squarer(out chan<- int, in <-chan int) {

for v := range in {

out <- v * v

}

close(out)

}

func printer(in <-chan int) {

for v := range in {

fmt.Println(v)

}

ptg16091132

}

func main() {

naturals := make(chan int)

squares := make(chan int)

go counter(naturals)

go squarer(squares, naturals)

printer(squares)

}

The cal l counter(naturals) implicitly converts naturals, a value of typ e chan int, to the type of the parameter, chan<- int. The printer(squares) call does a simi lar implicit conversion to <-chan int. Conversions fro m bidirec tion al to unidirec tion al channel typ es are permit ted in any assig nment. There is no going back, how ever: once you have a value of a unidirec tion al typ e such as chan<- int, there is no way to obt ain fro m it a value of typ e chan int that refers to the same channel dat a st ruc ture.

8.4.4. Buffered Channels

A buf fered channel has a queue of elements. The queue’s maximum size is deter mined when it is cre ate d, by the cap acity argument to make. The statement below cre ates a buf fered channel capable of holding three string values. Figure 8.2 is a graphic al repres entation of ch and the ch annel to which it refers.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

232

CHAPTER 8. GOROUTINES AND CHANNELS

ch = make(chan string, 3)

Figure 8.2. An emp ty buf fered channel.

A send operat ion on a buf fered channel ins erts an element at the back of the queue, and a re ceive operat ion remov es an element fro m the fro nt. If the channel is full, the send operat ion blocks its goroutine unt i l sp ace is made available by another goroutine’s receive . Conv ers ely, if the channel is emp ty, a receive operat ion blo cks until a value is sent by another goroutine.

We can send up to three values on this channel wit hout the goroutine blo cking: ch <- "A"

ch <- "B"

ch <- "C"

At this point, the channel is full (Figure 8.3), and a fourth send statement wou ld blo ck.

ptg16091132

Figure 8.3. A full buf fered channel.

If we receive one value,

fmt.Println(<-ch) // "A"

the channel is neither full nor emp ty (Figure 8.4), so eit her a send operat ion or a receive operat ion cou ld pro ceed wit hout blo cking . In this way, the channel’s buf fer decoup les the sending and receiving goro utines.

Figure 8.4. A par tially full buf fered channel.

In the unlikely event that a program needs to know the channel’s buf fer cap acity, it can be obtained by cal ling the bui lt-in cap func tion:

fmt.Println(cap(ch)) // "3"

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.4. CHANNELS

233

Wh en app lie d to a channel, the bui lt-in len func tion retur ns the number of elements current ly buf fered. Since in a conc urrent program this infor mat ion is likely to be stale as soon as it is ret rie ved, its value is limite d, but it cou ld con ceivably be useful dur ing fau lt diagnosis or perfor mance opt imizat ion.

fmt.Println(len(ch)) // "2"

Af ter two more receive operat ions the channel is emp ty again, and a fourth wou ld blo ck: fmt.Println(<-ch) // "B"

fmt.Println(<-ch) // "C"

In this example, the send and receive operat ions were all per for med by the same goroutine, but in real programs the y are usu ally exe cut ed by dif ferent goroutines. Nov ices are som etimes temp ted to use buf fered channel s within a single goroutine as a queue, lured by their ple asingly simple syntax, but this is a mistake . Channel s are deeply conne cte d to goroutine sche duling, and wit hout another goroutine receiving fro m the channel, a sender—and perhaps the whole program—risks becoming blo cke d fore ver. If all you need is a simple queue, make on e using a slice.

The example below shows an applic ation of a buf fered channel. It makes paral lel requests to thre e mi rro rs, that is, equivalent but geographic ally distr ibute d servers. It sends their resp ons es over a buf fered channel, then receives and retur ns only the firs t resp ons e, which is the quickest one to arrive . Thus mirroredQuery returns a result even before the two slower ptg16091132

servers have responded. (In cidentally, it’s quite nor mal for several goroutines to send values to the same channel con cur rently, as in this example, or to receive fro m the same channel.) func mirroredQuery() string {

responses := make(chan string, 3)

go func() { responses <- request("asia.gopl.io") }() go func() { responses <- request("europe.gopl.io") }() go func() { responses <- request("americas.gopl.io") }() return <-responses // return the quickest response

}

func request(hostname string) (response string) { /* ... */ }

Had we used an unbuf fered channel, the two slower goroutines wou ld have got ten stuck trying to send their respons es on a channel fro m which no goroutine will ever receive . This situation, cal le d a goro utine leak, wou ld be a bug . Un like garb age var iables, leaked goroutines are not aut omat ical ly col lec ted, so it is imp ortant to make sure that goroutines ter minate themselves when no lon g er needed.

The choice bet ween unbuf fered and buf fered channel s, and the choice of a buf fered channel’s capacity, may bot h af fec t the cor rec tness of a program. Unbuf fered channel s give stron g er sy nchro nizat ion guarante es because ever y send operat ion is synchro nize d with its corresp onding receive; wit h buffered channel s, these operat ions are decoupled. Als o, when we know an upp er bound on the number of values that will be sent on a channel, it’s not unu sual to cre ate a buf fered channel of that size and per for m al l the sends before the firs t value is re ceive d. Fai lure to allocate sufficient buf fer cap acity wou ld cause the program to deadlock.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

234

CHAPTER 8. GOROUTINES AND CHANNELS

Channel buf fer ing may also affec t prog ram perfor mance. Imagine three cooks in a cake shop, on e baking , on e icing, and one ins cribing each cake before passing it on to the next cook in the assemb l y line. In a kit c hen wit h litt le space, each cook that has finishe d a cake must wait for the next cook to becom e re ady to accept it; this rende zvou s is analogou s to communic ation ov er an unbuf fered channel.

If there is space for one cake bet ween each cook, a cook may place a finishe d cake there and immediate ly start wor k on the next; this is analogou s to a buf fered channel wit h capacity 1. So long as the cooks wor k at about the same rate on average, most of these handovers pro ceed quickly, smo othing out transient dif ferences in their respective rates. More space bet ween co oks—larger buf fers—can smo oth out bigger transient var iat ions in their rates wit hout st alling the ass emb l y line, such as happens when one cook takes a short bre ak, then later rushes to catch up.

On the other hand, if an earlier stage of the ass emb l y line is con sistent ly faster than the fol lowing stage , the buf fer bet ween them will spend most of its time full. Convers ely, if the later st age is faster, the buf fer will usu ally be emp ty. A buffer provides no benefit in this cas e.

The ass emb l y line met aphor is a useful one for channel s and goroutines. For example, if the second stage is more elaborate, a single cook may not be able to keep up wit h the sup ply fro m the firs t co ok or meet the demand fro m the third. To solve the pro blem, we cou ld hire another co ok to help the secon d, per for ming the same task but wor king indep endently. This is analo-ptg16091132

gous to cre ating another goroutine communic ating over the same channel s.

We don’t have space to show it here, but the gopl.io/ch8/cake package simulates this cake shop, wit h several parameters you can vary. It includes benchmarks (§11.4) for a few of the scenar ios des crib ed above .

8.5. Looping in Parallel

In this sec tion, we’ll explore som e common con cur rency patterns for exe cut ing all the iteration s of a loop in paral lel. We’ll con sider the pro blem of pro ducing thumbnail-size images from a set of full-size ones. The gopl.io/ch8/thumbnail package provides an ImageFile func tion that can scale a single image . We won’t show its implementation but it can be down loade d from gopl.io.

gopl.io/ch8/thumbnail

package thumbnail

// ImageFile reads an image from infile and writes

// a thumbnail-size version of it in the same directory.

// It returns the generated file name, e.g., "foo.thumb.jpg".

func ImageFile(infile string) (string, error)

The program below loops over a list of image file names and pro duces a thumbnail for each on e:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.5. LOOPING IN PARALLEL

235

gopl.io/ch8/thumbnail

// makeThumbnails makes thumbnails of the specified files.

func makeThumbnails(filenames []string) {

for _, f := range filenames {

if _, err := thumbnail.ImageFile(f); err != nil {

log.Println(err)

}

}

}

Obviou sly the order in which we pro cess the files doesn’t matter, since each scaling operat ion is indep endent of all the others. Pro blems like this that con sist ent ire ly of subproblems that are completely indep endent of each other are des crib ed as embarrassingly paral lel. Emb arrassingly paral lel pro blems are the easiest kind to imp lement con cur rently and enj oy per for mance that scales linearly wit h the amount of paral lelism.

Let’s exe cut e al l thes e op erat ions in paral lel, there by hiding the latency of the file I/O and using multiple CPUs for the image-s caling computation s. Our firs t attemp t at a conc urrent version just adds a go ke yword . We’ll ignore error s for now and address them later.

// NOTE: incorrect!

func makeThumbnails2(filenames []string) {

for _, f := range filenames {

ptg16091132

go thumbnail.ImageFile(f) // NOTE: ignoring errors

}

}

This version runs really fast—to o fast, in fact, since it takes less time than the original, even when the slice of file names cont ains only a single element. If there’s no paral lelism, how can the con cur rent version possibly run faster? The answer is that makeThumbnails returns before it has finishe d doing what it was sup pos ed to do. It starts all the goroutines, one per file name, but doesn’t wait for them to finish.

There is no direc t way to wait unt i l a goroutine has finishe d, but we can change the inner goro utine to rep ort its comp let ion to the out er goroutine by sending an event on a share d ch annel. Since we know that there are exac tly len(filenames) inner goroutines, the out er goro utine need only count that many events before it retur ns:

// makeThumbnails3 makes thumbnails of the specified files in parallel.

func makeThumbnails3(filenames []string) {

ch := make(chan struct{})

for _, f := range filenames {

go func(f string) {

thumbnail.ImageFile(f) // NOTE: ignoring errors

ch <- struct{}{}

}(f)

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

236

CHAPTER 8. GOROUTINES AND CHANNELS

// Wait for goroutines to complete.

for range filenames {

<-ch

}

}

No tice that we passed the value of f as an explicit argument to the lit eral function ins tead of using the declarat ion of f from the enclosing for lo op: for _, f := range filenames {

go func() {

thumbnail.ImageFile(f) // NOTE: incorrect!

// ...

}()

}

Re call the pro blem of loop var iable capture inside an anonymou s func tion, descr ibed in Section 5.6.1. Above , the single var iable f is share d by all the anony mou s func tion values and up dated by successive loop iterat ions. By the time the new goroutines start exe cut ing the literal function, the for lo op may have updated f and started another iterat ion or (more likely) finishe d entire ly, so when these goroutines read the value of f, the y al l observe it to have the value of the final element of the slice. By adding an explicit parameter, we ens ure that we use the value of f that is cur rent when the go st atement is exe cut ed.

Wh at if we want to retur n values fro m each wor ker goroutine to the main one? If the cal l to ptg16091132

thumbnail.ImageFile fai ls to cre ate a file, it retur ns an error. The next version of makeThumbnails returns the firs t er ror it receives fro m any of the scaling operat ions:

// makeThumbnails4 makes thumbnails for the specified files in parallel.

// It returns an error if any step failed.

func makeThumbnails4(filenames []string) error {

errors := make(chan error)

for _, f := range filenames {

go func(f string) {

_, err := thumbnail.ImageFile(f)

errors <- err

}(f)

}

for range filenames {

if err := <-errors; err != nil {

return err // NOTE: incorrect: goroutine leak!

}

}

return nil

}

This function has a subtle bug . Wh en it encount ers the firs t non-ni l er ror, it retur ns the error to the cal ler, leaving no goroutine draining the errors ch annel. Each remaining wor ker goro utine will blo ck forever when it tries to send a value on that channel, and will never

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.5. LOOPING IN PARALLEL

237

terminate. This sit uat ion, a goroutine leak (§8.4.4), may cause the whole program to get stuck or to run out of memor y.

The simplest solut ion is to use a buf fered channel wit h sufficient cap acity that no wor ker goro utine will blo ck when it sends a message . (An alt ernat ive solut ion is to cre ate another goro utine to drain the channel whi le the main goroutine retur ns the firs t er ror wit hout del ay.) The next version of makeThumbnails us es a buf fered channel to retur n the names of the generated image files along wit h any error s.

// makeThumbnails5 makes thumbnails for the specified files in parallel.

// It returns the generated file names in an arbitrary order,

// or an error if any step failed.

func makeThumbnails5(filenames []string) (thumbfiles []string, err error) {

type item struct {

thumbfile string

err

error

}

ch := make(chan item, len(filenames))

for _, f := range filenames {

go func(f string) {

var it item

ptg16091132

it.thumbfile, it.err = thumbnail.ImageFile(f)

ch <- it

}(f)

}

for range filenames {

it := <-ch

if it.err != nil {

return nil, it.err

}

thumbfiles = append(thumbfiles, it.thumbfile)

}

return thumbfiles, nil

}

Our final version of makeThumbnails, below, retur ns the tot al number of bytes occ upied by the new files. Unlike the pre vious versions, how ever, it receives the file names not as a slice but ov er a channel of str ings, so we cannot pre dic t the number of loop iterat ions.

To know when the last goroutine has finishe d (w hich may not be the last one to start), we need to increment a count er before each goroutine starts and decrement it as each goroutine finishes. This demands a speci al kind of count er, one that can be safely manipu lated fro m mu ltiple goroutines and that provides a way to wait unt i l it becom es zero. This count er typ e is know n as sync.WaitGroup, and the code below shows how to use it:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

238

CHAPTER 8. GOROUTINES AND CHANNELS

// makeThumbnails6 makes thumbnails for each file received from the channel.

// It returns the number of bytes occupied by the files it creates.

func makeThumbnails6(filenames <-chan string) int64 {

sizes := make(chan int64)

var wg sync.WaitGroup // number of working goroutines

for f := range filenames {

wg.Add(1)

// worker

go func(f string) {

defer wg.Done()

thumb, err := thumbnail.ImageFile(f)

if err != nil {

log.Println(err)

return

}

info, _ := os.Stat(thumb) // OK to ignore error

sizes <- info.Size()

}(f)

}

// closer

go func() {

wg.Wait()

close(sizes)

}()

ptg16091132

var total int64

for size := range sizes {

total += size

}

return total

}

No te the asy mmet ry in the Add and Done methods. Add, which increments the count er, must be cal le d before the wor ker goroutine starts, not wit hin it; other wis e we would not be sure that the Add happ ens before the ‘‘clos er’’ goro utine cal ls Wait. Als o, Add takes a parameter, but Done do es not; it’s equivalent to Add(-1). We use defer to ens ure that the count er is decrement ed even in the error cas e. The str ucture of the code above is a common and idiomatic pattern for lo oping in paral lel when we don’t know the number of iterat ions.

The sizes ch annel car r ies each file size back to the main goroutine, which receives them using a range lo op and computes the sum. Obs erve how we cre ate a clos er goro utine that waits for the wor kers to finish before closing the sizes ch annel. These two operat ions, wait and clos e, must be con cur rent wit h the loop over sizes. Con sider the alt ernat ives: if the wait op erat ion were place d in the main goroutine before the loop, it wou ld never end, and if place d af ter the loop, it wou ld be unreach able since wit h nothing closing the channel, the loop wou ld ne ver ter minate.

Figure 8.5 illustrates the sequence of events in the makeThumbnails6 func tion. The ver tic al lines represent goroutines. The thin seg ments indic ate sle ep, the thick seg ments activity. The

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.6. EXAMPLE: CONCURRENT WEB CRAWLER

239

Figure 8.5. The sequence of events in makeThumbnails6.

ptg16091132

di agon al arrows indic ate events that synchro nize one goroutine wit h anot her. Time flows down. Not ice how the main goroutine spend s most of its time in the range lo op asleep, waiting for a wor ker to send a value or the clos er to clos e the channel.

Exercis e 8.4: Mo dif y the reverb2 server to use a sync.WaitGroup per conne ction to count the number of active echo goro utines. When it fal ls to zero, clos e the write half of the TCP

connec tion as des crib ed in Exercis e 8.3. Ver ify that your modified netcat3 client fro m that exercise waits for the final ech oes of multiple con cur rent shouts, even after the stand ard inp ut has been clos ed.

Exercis e 8.5: Take an exist ing CPU-b ound sequential program, such as the Mandelbro t prog ram of Sec tion 3.3 or the 3-D sur face comp utation of Sec tion 3.2, and exe cut e its main lo op in paral lel using channel s for communic ation. How much faster does it run on a mu ltiprocessor machine? What is the opt imal number of goroutines to use?

8.6. Example: Concurrent Web Crawler

In Sec tion 5.6, we made a simple web craw ler that explore d the lin k graph of the web in breadt h-firs t order. In this sec tion, we’ll make it con cur rent so that indep endent cal ls to crawl can exploit the I/O paral lelism available in the web. The crawl func tion remains exac tly as it was in gopl.io/ch5/findlinks3:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

240

CHAPTER 8. GOROUTINES AND CHANNELS

gopl.io/ch8/crawl1

func crawl(url string) []string {

fmt.Println(url)

list, err := links.Extract(url)

if err != nil {

log.Print(err)

}

return list

}

The main function res embles breadthFirst (§5.6). As before, a wor klist records the queue of it ems that need pro cessing, each item being a list of URLs to craw l, but this time, ins tead of repres enting the queue using a slice, we use a channel. Each cal l to crawl occurs in its own goro utine and sends the lin ks it dis cov ers back to the wor klist.

func main() {

worklist := make(chan []string)

// Start with the command-line arguments.

go func() { worklist <- os.Args[1:] }()

// Crawl the web concurrently.

seen := make(map[string]bool)

ptg16091132

for list := range worklist {

for _, link := range list {

if !seen[link] {

seen[link] = true

go func(link string) {

worklist <- crawl(link)

}(link)

}

}

}

}

No tice that the craw l goro utine takes link as an explicit parameter to avoid the pro blem of lo op var iable capture we firs t saw in Sec tion 5.6.1. Als o notice that the initial send of the command-line arguments to the wor klist must run in its own goroutine to avoid dead lock, a stuck situ ation in which bot h the main goroutine and a craw ler goro utine att emp t to send to each ot her whi le neit her is receiving . An alt ernat ive solut ion wou ld be to use a buf fered channel.

The craw ler is now hig hly con cur rent and prints a storm of URLs, but it has two pro blems.

The firs t problem manifests its elf as erro r mess ages in the log after a few second s of operat ion: $ go build gopl.io/ch8/crawl1

$ ./crawl1 http://gopl.io/

http://gopl.io/

https://golang.org/help/

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.6. EXAMPLE: CONCURRENT WEB CRAWLER

241

https://golang.org/doc/

https://golang.org/blog/

...

2015/07/15 18:22:12 Get ...: dial tcp: lookup blog.golang.org: no such host 2015/07/15 18:22:12 Get ...: dial tcp 23.21.222.120:443: socket: too many open files

...

The initial erro r mess age is a sur prising rep ort of a DNS lookup fai lure for a reliable dom ain.

The subsequent error message reveals the cause: the program create d so many net wor k conne ction s at once that it exceeded the per-pro cess limit on the number of open files, causing op erat ions such as DNS lookups and cal ls to net.Dial to start fai ling.

The program is to o paral lel. Unb ounde d paral lelism is rarely a good ide a since there is always a limiting fac tor in the system, such as the number of CPU cores for compute-b ound work loads, the number of spind les and heads for local disk I/O operat ions, the bandw idt h of the net wor k for streaming dow nlo ads, or the ser ving cap acity of a web ser vice. The solut ion is to limit the number of paral lel uses of the res ource to match the level of paral lelism that is avai lable. A simple way to do that in our example is to ens ure that no more than n calls to links.Extract are active at once, where n is comfortably less than the file descr ipt or limit—20, say. This is analogou s to the way a door man at a crow ded nig htclub admits a guest on ly when som e ot her guest leaves.

ptg16091132

We can limit paral lelism using a buf fered channel of cap acity n to model a con cur rency primi-tive cal le d a counting semap hore. Con ceptu ally, each of the n vac ant slots in the channel buf fer repres ents a token entitling the holder to pro ceed. Sending a value into the channel acquires a token, and receiving a value fro m the channel releases a token, creating a new vac ant slot.

This ens ures that at most n send s can occ ur without an int ervening receive . (A lthough it mig ht be more int uit ive to tre at fill ed slots in the channel buf fer as tokens, using vac ant slots avoids the need to fill the channel buf fer after cre ating it.) Since the channel element typ e is not imp ortant, we’ll use struct{}, which has size zero.

Let’s rewrite the crawl func tion so that the cal l to links.Extract is brackete d by operat ions to acquire and release a token, thu s ensuring that at most 20 cal ls to it are active at one time.

It’s good prac tice to keep the semaphore operat ions as clos e as possible to the I/O operat ion they regu late.

gopl.io/ch8/crawl2

// tokens is a counting semaphore used to

// enforce a limit of 20 concurrent requests.

var tokens = make(chan struct{}, 20)

func crawl(url string) []string {

fmt.Println(url)

tokens <- struct{}{} // acquire a token

list, err := links.Extract(url)

<-tokens // release the token

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

242

CHAPTER 8. GOROUTINES AND CHANNELS

if err != nil {

log.Print(err)

}

return list

}

The secon d problem is that the program never ter minates, even when it has discov ere d al l the lin ks re ach able fro m the initial URLs. (Of course, you’re unlikely to not ice this pro blem unless you cho ose the initial URLs caref ully or imp lement the depth-limiting feature of Exercis e 8.6.) Fo r the program to ter minate, we need to bre ak out of the main loop when the wor klist is empt y an d no craw l goro utines are active.

func main() {

worklist := make(chan []string)

var n int // number of pending sends to worklist

// Start with the command-line arguments.

n++

go func() { worklist <- os.Args[1:] }()

// Crawl the web concurrently.

seen := make(map[string]bool)

for ; n > 0; n-- {

list := <-worklist

ptg16091132

for _, link := range list {

if !seen[link] {

seen[link] = true

n++

go func(link string) {

worklist <- crawl(link)

}(link)

}

}

}

}

In this version, the count er n ke eps track of the number of sends to the wor klist that are yet to occur. Each time we know that an item needs to be sent to the wor klist, we increment n, once before we send the initial command-line arguments, and again each time we start a craw ler goro utine. The main loop ter minates when n fal ls to zero, since there is no more wor k to be done.

No w the con cur rent craw ler runs about 20 times faster than the bre adt h-firs t craw ler from Section 5.6, wit hout error s, and ter minates correc tly if it should complete its task.

The program below shows an alternat ive solut ion to the pro blem of excessive con cur rency.

This version uses the original crawl func tion that has no count ing semaphore , but cal ls it from one of 20 lon g-live d craw ler goro utines, thu s ensuring that at most 20 HTTP requests are ac tive con cur rently.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.6. EXAMPLE: CONCURRENT WEB CRAWLER

243

gopl.io/ch8/crawl3

func main() {

worklist := make(chan []string)

// lists of URLs, may have duplicates

unseenLinks := make(chan string) // de-duplicated URLs

// Add command-line arguments to worklist.

go func() { worklist <- os.Args[1:] }()

// Create 20 crawler goroutines to fetch each unseen link.

for i := 0; i < 20; i++ {

go func() {

for link := range unseenLinks {

foundLinks := crawl(link)

go func() { worklist <- foundLinks }()

}

}()

}

// The main goroutine de-duplicates worklist items

// and sends the unseen ones to the crawlers.

seen := make(map[string]bool)

for list := range worklist {

for _, link := range list {

if !seen[link] {

seen[link] = true

ptg16091132

unseenLinks <- link

}

}

}

}

The craw ler goro utines are all fed by the same channel, unseenLinks. The main goroutine is resp onsible for de-dup lic ating items it receives fro m the wor klist, and then sending each unseen one over the unseenLinks ch annel to a craw ler goro utine.

The seen map is confine d within the main goroutine; that is, it can be accessed only by that goro utine. Like other for ms of infor mat ion hiding, confinement helps us reason about the correc tness of a program. For example, local variables cannot be mention ed by name fro m outside the function in which the y are declare d; var iables that do not escap e (§2.3.4) fro m a func tion cannot be accessed fro m outside that function; and enc apsulated fields of an obj e ct cannot be accessed except by the methods of that obj e ct. In all cas es, infor mat ion hiding helps to limit unintended int erac tions bet ween par ts of the program.

Links found by crawl are sent to the wor klist fro m a dedic ated goroutine to avoid dead lock.

To save space, we have not addressed the pro blem of ter minat ion in this example.

Exercis e 8.6: Add depth-limiting to the con cur rent craw ler. That is, if the user sets -depth=3, then only URLs reach able by at most three lin ks wi l l be fetch ed.

Exercis e 8.7: Wr ite a con cur rent program that cre ates a local mirro r of a web sit e , fetching each reach able page and writing it to a direc tor y on the local disk. Only pages wit hin the

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

244

CHAPTER 8. GOROUTINES AND CHANNELS

or iginal dom ain (for ins tance, golang.org) should be fetch ed. URLs wit hin mirro red pages should be alt ere d as needed so that the y refer to the mir rored page , not the original.

8.7. Multiplexing with select

The program below does the count dow n for a rocket launch. The time.Tick func tion retur ns a channel on which it sends events per iodic ally, acting like a met ron om e. The value of each event is a timestamp, but it is rarely as int erest ing as the fac t of its deliver y.

gopl.io/ch8/countdown1

func main() {

fmt.Println("Commencing countdown.")

tick := time.Tick(1 * time.Second)

for countdown := 10; countdown > 0; countdown-- {

fmt.Println(countdown)

<-tick

}

launch()

}

No w let’s add the abi lit y to abort the launch sequence by pressing the retur n ke y during the ptg16091132

countdow n. First, we start a goroutine that tries to read a single byte fro m the stand ard inp ut and, if it succe e ds, sends a value on a channel cal le d abort.

gopl.io/ch8/countdown2

abort := make(chan struct{})

go func() {

os.Stdin.Read(make([]byte, 1)) // read a single byte

abort <- struct{}{}

}()

No w each iterat ion of the count dow n lo op needs to wait for an event to arrive on one of the two channel s: the ticker channel if everything is fine (‘‘nominal’’ in NASA jargon) or an abort event if there was an ‘‘anom aly.’’ We can’t just receive fro m each channel because whiche ver op erat ion we try firs t wi l l block unt i l comp let ion. We need to mu ltipl ex thes e op erat ions, and to do that, we need a select statement.

select {

case <-ch1:

// ...

case x := <-ch2:

// ...use x...

case ch3 <- y:

// ...

default:

// ...

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.7. MULTIPLEXING WITH SELECT

245

The general for m of a selec t st atement is shown above . Like a switch statement, it has a number of cas es and an opt ion al default. Each cas e sp ecifies a commu nicati on (a send or receive op erat ion on som e ch annel) and an ass oci ated blo ck of statements. A re ceive expression may appear on its own, as in the firs t case, or wit hin a short var iable decl arat ion, as in the secon d case; the secon d form lets you refer to the receive d value.

A select waits until a communic ation for som e case is ready to pro ceed. It then per for ms that communic ation and exe cut es the cas e’s ass oci ated statements; the other communic ations do not happen. A select with no cas es, select{}, waits forever.

Let’s retur n to our rocket launch program. The time.After func tion immediate ly retur ns a ch annel, and starts a new goroutine that sends a single value on that channel after the specified time. The selec t st atement below waits until the firs t of two events arrives, either an abor t event or the event indic ating that 10 secon ds have elaps ed. If 10 secon ds go by wit h no abort, the launch pro ceeds.

func main() {

// ...create abort channel...

fmt.Println("Commencing countdown.

Press return to abort.")

select {

case <-time.After(10 * time.Second):

// Do nothing.

case <-abort:

ptg16091132

fmt.Println("Launch aborted!")

return

}

launch()

}

The example below is more subtle. The channel ch, whose buf fer size is 1, is alt ernately emp ty then full, so only one of the cas es can pro ceed, eit her the send when i is even, or the receive when i is odd. It always prints 0 2 4 6 8.

ch := make(chan int, 1)

for i := 0; i < 10; i++ {

select {

case x := <-ch:

fmt.Println(x) // "0" "2" "4" "6" "8"

case ch <- i:

}

}

If multiple cas es are ready, select picks one at random, which ens ures that every channel has an equ al chance of being selec ted. Incre asing the buf fer size of the pre vious example makes its output non deter minist ic, because when the buf fer is neither full nor emp ty, the selec t st atement figurat ive ly tosses a coin.

Let’s make our launch program print the count dow n. The selec t st atement below causes each it erat ion of the loop to wait up to 1 secon d for an abort, but no lon g er.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

246

CHAPTER 8. GOROUTINES AND CHANNELS

gopl.io/ch8/countdown3

func main() {

// ...create abort channel...

fmt.Println("Commencing countdown.

Press return to abort.")

tick := time.Tick(1 * time.Second)

for countdown := 10; countdown > 0; countdown-- {

fmt.Println(countdown)

select {

case <-tick:

// Do nothing.

case <-abort:

fmt.Println("Launch aborted!")

return

}

}

launch()

}

The time.Tick func tion beh aves as if it cre ates a goroutine that cal ls time.Sleep in a loop, sending an event each time it wakes up. When the count dow n func tion above retur ns, it stops re ceiving events fro m tick, but the ticker goroutine is still there , tr ying in vain to send on a ch annel fro m which no goroutine is receiving—a goro utine leak (§8.4.4).

ptg16091132

The Tick func tion is convenient, but it’s appro priate only when the ticks will be needed thro ughout the lifet ime of the app lic ation. Other wis e, we should use this pattern: ticker := time.NewTicker(1 * time.Second)

<-ticker.C // receive from the ticker's channel

ticker.Stop() // cause the ticker's goroutine to terminate Sometimes we want to try to send or receive on a channel but avoid blo cking if the channel is not ready—a non-blocki n g communic ation. A sele ct statement can do that too. A select may have a default, which specifies what to do when non e of the other communic ations can proceed immediate ly.

The selec t st atement below receives a value fro m the abort ch annel if there is one to receive; ot her wis e it does not hing. This is a non-blo cking receive operat ion; doing it rep eatedly is called polling a channel.

select {

case <-abort:

fmt.Printf("Launch aborted!\n")

return

default:

// do nothing

}

The zero value for a channel is nil. Perhaps sur prisingly, nil channel s are som etimes useful.

Becaus e send and receive operat ions on a nil channel blo ck forever, a cas e in a selec t st atement

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.8. EXAMPLE: CONCURRENT DIRECTORY TRAVERSAL

247

whos e ch annel is nil is never selec ted. This lets us use nil to enable or dis able cas es that corresp ond to features like handling timeouts or cancellat ion, responding to other input events, or emitt ing out put. We’ll see an example in the next sec tion.

Exercis e 8.8: Using a selec t st atement, add a timeout to the ech o server fro m Section 8.3 so that it dis conne cts any client that shouts not hing wit hin 10 secon ds.

8.8. Example: Concurrent Directory Traversal

In this sec tion, we’ll bui ld a program that rep orts the disk usage of one or more direc tor ies sp ecified on the command line, like the Unix du command. Most of its wor k is don e by the walkDir func tion below, which enumerates the ent ries of the direc tor y dir using the dirents helper function.

gopl.io/ch8/du1

// walkDir recursively walks the file tree rooted at dir

// and sends the size of each found file on fileSizes.

func walkDir(dir string, fileSizes chan<- int64) {

for _, entry := range dirents(dir) {

if entry.IsDir() {

subdir := filepath.Join(dir, entry.Name())

ptg16091132

walkDir(subdir, fileSizes)

} else {

fileSizes <- entry.Size()

}

}

}

// dirents returns the entries of directory dir.

func dirents(dir string) []os.FileInfo {

entries, err := ioutil.ReadDir(dir)

if err != nil {

fmt.Fprintf(os.Stderr, "du1: %v\n", err)

return nil

}

return entries

}

The ioutil.ReadDir func tion retur ns a slice of os.FileInfo—the same infor mat ion that a call to os.Stat returns for a single file. For each sub direc tor y, walkDir re cursive ly cal ls its elf, and for each file, walkDir send s a message on the fileSizes ch annel. The mess age is the size of the file in bytes.

The main function, shown below, uses two goro utines. The backg round goroutine cal ls walkDir for each direc tor y sp ecified on the command line and finally clos es the fileSizes ch annel. The main goroutine computes the sum of the file sizes it receives fro m the channel and finally prints the tot al.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

248

CHAPTER 8. GOROUTINES AND CHANNELS

// The du1 command computes the disk usage of the files in a directory.

package main

import (

"flag"

"fmt"

"io/ioutil"

"os"

"path/filepath"

)

func main() {

// Determine the initial directories.

flag.Parse()

roots := flag.Args()

if len(roots) == 0 {

roots = []string{"."}

}

// Traverse the file tree.

fileSizes := make(chan int64)

go func() {

for _, root := range roots {

walkDir(root, fileSizes)

}

ptg16091132

close(fileSizes)

}()

// Print the results.

var nfiles, nbytes int64

for size := range fileSizes {

nfiles++

nbytes += size

}

printDiskUsage(nfiles, nbytes)

}

func printDiskUsage(nfiles, nbytes int64) {

fmt.Printf("%d files

%.1f GB\n", nfiles, float64(nbytes)/1e9)

}

This program pauses for a lon g while before print ing its result: $ go build gopl.io/ch8/du1

$ ./du1 $HOME /usr /bin /etc

213201 files

62.7 GB

The program wou ld be nicer if it kept us infor med of its progress. How ever, simply mov ing the printDiskUsage call int o the loop wou ld cause it to print thous and s of lines of out put.

The var iant of du below prints the tot als per iodic ally, but only if the -v flag is specified since not all users will want to see progress messages. The backg round goroutine that loops over roots remains unchange d. The main goroutine now uses a ticker to generate events every

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.8. EXAMPLE: CONCURRENT DIRECTORY TRAVERSAL

249

500ms, and a selec t st atement to wait for eit her a file size message , in which cas e it updates the totals, or a tick event, in which cas e it prints the cur rent tot als. If the -v flag is not specified, the tick ch annel remains nil, and its cas e in the select is effec tive ly dis abled.

gopl.io/ch8/du2

var verbose = flag.Bool("v", false, "show verbose progress messages") func main() {

// ...start background goroutine...

// Print the results periodically.

var tick <-chan time.Time

if *verbose {

tick = time.Tick(500 * time.Millisecond)

}

var nfiles, nbytes int64

loop:

for {

select {

case size, ok := <-fileSizes:

if !ok {

break loop // fileSizes was closed

}

nfiles++

nbytes += size

ptg16091132

case <-tick:

printDiskUsage(nfiles, nbytes)

}

}

printDiskUsage(nfiles, nbytes) // final totals

}

Since the program no lon g er uses a range lo op, the firs t select case must explicitly test whet her the fileSizes ch annel has been clos ed, using the two-result for m of receive operation. If the channel has been clos ed, the program bre aks out of the loop. The lab ele d break st atement bre aks out of bot h the select and the for lo op; an unlabele d break would bre ak out of only the select, causing the loop to beg in the next iterat ion.

The program now gives us a leisure ly stream of updates: $ go build gopl.io/ch8/du2

$ ./du2 -v $HOME /usr /bin /etc

28608 files

8.3 GB

54147 files

10.3 GB

93591 files

15.1 GB

127169 files

52.9 GB

175931 files

62.2 GB

213201 files

62.7 GB

Ho wever, it still takes too lon g to finish. There’s no reason why all the cal ls to walkDir can’t be done con cur rently, there by exploit ing paral lelism in the disk system. The third version of du,

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

250

CHAPTER 8. GOROUTINES AND CHANNELS

below, cre ates a new goroutine for each cal l to walkDir. It uses a sync.WaitGroup (§8.5) to count the number of cal ls to walkDir that are still active, and a clos er goro utine to clos e the fileSizes ch annel when the count er dro ps to zero.

gopl.io/ch8/du3

func main() {

// ...determine roots...

// Traverse each root of the file tree in parallel.

fileSizes := make(chan int64)

var n sync.WaitGroup

for _, root := range roots {

n.Add(1)

go walkDir(root, &n, fileSizes)

}

go func() {

n.Wait()

close(fileSizes)

}()

// ...select loop...

}

func walkDir(dir string, n *sync.WaitGroup, fileSizes chan<- int64) {

defer n.Done()

ptg16091132

for _, entry := range dirents(dir) {

if entry.IsDir() {

n.Add(1)

subdir := filepath.Join(dir, entry.Name())

go walkDir(subdir, n, fileSizes)

} else {

fileSizes <- entry.Size()

}

}

}

Since this program creates many thous and s of goroutines at its peak, we have to change dirents to use a count ing semaphore to pre vent it fro m op ening too many files at once, just as we did for the web craw ler in Sec tion 8.6:

// sema is a counting semaphore for limiting concurrency in dirents.

var sema = make(chan struct{}, 20)

// dirents returns the entries of directory dir.

func dirents(dir string) []os.FileInfo {

sema <- struct{}{}

// acquire token

defer func() { <-sema }() // release token

// ...

This version runs several times faster than the pre vious one, thoug h there is a lot of var iabilit y from system to system.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.9. CANCELLATION

251

Exercis e 8.9: Wr ite a version of du that comp utes and per iodic ally displ ays sep arate tot als for each of the root direc tor ies.

8.9. Cancellation

Sometimes we need to ins truct a goroutine to stop what it is doing , for example, in a web server per for ming a computation on beh alf of a client that has disconne cte d.

There is no way for one goroutine to ter minate another direc tly, since that wou ld leave all its sh are d var iables in undefine d st ates. In the rocket launch program (§8.7) we sent a single value on a channel named abort, which the count dow n goro utine int erprete d as a request to stop its elf. But what if we need to cancel two goro utines, or an arbit rar y numb er?

One possibi lit y mig ht be to send as many events on the abort ch annel as there are goroutines to cancel. If som e of the goroutines have already ter minated thems elves, how ever, our count wi l l be too large , and our sends will get stuck. On the other hand, if those goroutines have sp awned other goroutines, our count will be too small, and som e goro utines will remain unaware of the cancellat ion. In general, it’s hard to know how many goroutines are wor king on our beh alf at any given mom ent. Moreover, when a goroutine receives a value fro m the abort ch annel, it con sumes that value so that other goroutines won’t see it. For cancellat ion, what we need is a re liable mechanism to broadcast an event over a channel so that many ptg16091132

goro utines can see it as it occ urs and can later see that it has occurred.

Re call that after a channel has been clos ed and drained of all sent values, subsequent receive op erat ions pro ceed immediate ly, yielding zero values. We can exploit this to cre ate a bro adcast mechanism: don’t send a value on the channel, cl ose it.

We can add cancellat ion to the du prog ram from the pre vious sec tion wit h a few simple ch anges. First, we cre ate a cancellat ion channel on which no values are ever sent, but whose closure indic ates that it is time for the program to stop what it is doing . We als o define a ut i lit y func tion, cancelled, that che cks or poll s the cancellat ion state at the ins tant it is cal le d.

gopl.io/ch8/du4

var done = make(chan struct{})

func cancelled() bool {

select {

case <-done:

return true

default:

return false

}

}

Next, we cre ate a goroutine that will read fro m the stand ard inp ut, which is typic ally conne cte d to the ter minal. As soon as any inp ut is read (for ins tance, the user press es the retur n ke y), this goroutine bro adc asts the cancellat ion by closing the done ch annel.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

252

CHAPTER 8. GOROUTINES AND CHANNELS

// Cancel traversal when input is detected.

go func() {

os.Stdin.Read(make([]byte, 1)) // read a single byte

close(done)

}()

No w we need to make our goroutines respond to the cancellat ion. In the main goroutine, we add a third cas e to the selec t st atement that tries to receive fro m the done ch annel. The function retur ns if this cas e is ever selec ted, but before it retur ns it must firs t drain the fileSizes ch annel, dis carding all values until the channel is clos ed. It does this to ens ure that any active calls to walkDir can run to complet ion wit hout getting stuck sending to fileSizes.

for {

select {

case <-done:

// Drain fileSizes to allow existing goroutines to finish.

for range fileSizes {

// Do nothing.

}

return

case size, ok := <-fileSizes:

// ...

}

ptg16091132

}

The walkDir goro utine pol ls the cancellat ion statu s when it beg ins, and retur ns wit hout doing anyt hing if the statu s is set. This tur ns all goroutines cre ate d af ter cancellat ion int o no-ops: func walkDir(dir string, n *sync.WaitGroup, fileSizes chan<- int64) {

defer n.Done()

if cancelled() {

return

}

for _, entry := range dirents(dir) {

// ...

}

}

It mig ht be profitable to pol l the cancellat ion statu s again wit hin walkDir’s loop, to avoid creat ing goroutines after the cancellat ion event. Cancellat ion inv olves a trade-off; a quicker resp ons e of ten requires more int rusive changes to program logic. Ensuring that no exp ensive op erat ions ever occ ur af ter the cancellat ion event may require updat ing many places in your co de, but often most of the benefit can be obt ained by che cking for cancellat ion in a few important places.

A lit tle profiling of this program reveale d that the bot tleneck was the acquisition of a semaph ore token in dirents. The select below makes this operat ion cancellable and reduces the typical cancellat ion latency of the program fro m hundre ds of millis econd s to tens:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.10. EXAMPLE: CHAT SERVER

253

func dirents(dir string) []os.FileInfo {

select {

case sema <- struct{}{}: // acquire token

case <-done:

return nil // cancelled

}

defer func() { <-sema }() // release token

// ...read directory...

}

No w, when cancellat ion occ urs, all the backg round goroutines quickly stop and the main function retur ns. Of course, when main returns, a program exits, so it can be hard to tel l a main func tion that cle ans up after its elf fro m on e that does not. There’s a handy trick we can use during testing: if ins tead of retur ning fro m main in the event of cancellat ion, we exe cut e a cal l to panic, then the runtime will dump the stack of every goroutine in the program. If the main goro utine is the only one lef t, then it has cle aned up after its elf. But if other goroutines remain, the y may not have been pro perly cancelled, or perhaps the y have been cancelled but the cancellat ion takes time; a litt le inv est igat ion may be wor thw hile. The panic dump often cont ains sufficient infor mat ion to distinguish these cas es.

Exercis e 8.10: HT TP re quests may be cancelled by closing the opt ion al Cancel ch annel in the ptg16091132

http.Request st ruc t. Mo dif y the web craw ler of Sec tion 8.6 to sup por t cancel lat ion.

Hint: the http.Get conv enience function does not give you an opp ortunity to customize a Request. Ins tead, cre ate the request using http.NewRequest, set its Cancel field, then perform the request by cal ling http.DefaultClient.Do(req).

Exercis e 8.11: Fo llowing the appro ach of mirroredQuery in Sec tion 8.4.4, implement a var iant of fetch that requests several URLs conc urrently. As soon as the firs t resp ons e ar r ives, cancel the other requests.

8.10. Example: Chat Server

We’ll finish this chapt er wit h a chat ser ver that lets several users bro adc ast textual messages to each other. There are four kinds of goroutine in this program. There is one ins tance apie ce of the main and broadcaster goro utines, and for each client conne ction there is one handleConn and one clientWriter goro utine. The bro adc aster is a good illustrat ion of how select is used, since it has to respond to three dif ferent kinds of messages.

The job of the main goroutine, shown below, is to listen for and accept incoming net wor k conne ction s from clients. For each one, it cre ates a new handleConn goro utine, just as in the concurrent ech o server we saw at the start of this chapt er.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

254

CHAPTER 8. GOROUTINES AND CHANNELS

gopl.io/ch8/chat

func main() {

listener, err := net.Listen("tcp", "localhost:8000") if err != nil {

log.Fatal(err)

}

go broadcaster()

for {

conn, err := listener.Accept()

if err != nil {

log.Print(err)

continue

}

go handleConn(conn)

}

}

Next is the bro adc aster. Its local var iable clients re cords the cur rent set of conne cte d clients.

The only infor mat ion recorde d ab out each client is the identity of its out going message channel, about which more later.

type client chan<- string // an outgoing message channel var (

ptg16091132

entering = make(chan client)

leaving = make(chan client)

messages = make(chan string) // all incoming client messages

)

func broadcaster() {

clients := make(map[client]bool) // all connected clients for {

select {

case msg := <-messages:

// Broadcast incoming message to all

// clients' outgoing message channels.

for cli := range clients {

cli <- msg

}

case cli := <-entering:

clients[cli] = true

case cli := <-leaving:

delete(clients, cli)

close(cli)

}

}

}

The bro adc aster listens on the global entering and leaving ch annel s for announcements of ar r iving and dep arting clients. When it receives one of these events, it updates the clients

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 8.10. EXAMPLE: CHAT SERVER

255

set, and if the event was a depar ture, it clos es the client’s out going message channel. The broadc aster als o listens for events on the global messages ch annel, to which each client sends al l its incoming messages. When the bro adc aster receives one of these events, it bro adc asts the mess age to every conne cte d client.

No w let’s look at the per-client goroutines. The handleConn func tion cre ates a new out going mess age channel for its client and announces the arrival of this client to the bro adc aster over the entering ch annel. Then it reads every line of text fro m the client, sending each line to the broadc aster over the global incoming message channel, prefixing each message wit h the identity of its sender. Once there is not hing more to read fro m the client, handleConn announces the dep arture of the client over the leaving ch annel and clos es the conne ction.

func handleConn(conn net.Conn) {

ch := make(chan string) // outgoing client messages

go clientWriter(conn, ch)

who := conn.RemoteAddr().String()

ch <- "You are " + who

messages <- who + " has arrived"

entering <- ch

input := bufio.NewScanner(conn)

for input.Scan() {

ptg16091132

messages <- who + ": " + input.Text()

}

// NOTE: ignoring potential errors from input.Err()

leaving <- ch

messages <- who + " has left"

conn.Close()

}

func clientWriter(conn net.Conn, ch <-chan string) {

for msg := range ch {

fmt.Fprintln(conn, msg) // NOTE: ignoring network errors

}

}

In addition, handleConn creates a clientWriter goro utine for each client that receives messages bro adc ast to the client’s out going message channel and writes them to the client’s network conne ction. The client writer’s loop ter minates when the bro adc aster clos es the channel af ter receiving a leaving notification.

The displ ay below shows the ser ver in action wit h two clients in sep arate windows on the same computer, using netcat to chat:

$ go build gopl.io/ch8/chat

$ go build gopl.io/ch8/netcat3

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

256

CHAPTER 8. GOROUTINES AND CHANNELS

$ ./chat &

$ ./netcat3

You are 127.0.0.1:64208

$ ./netcat3

127.0.0.1:64211 has arrived

You are 127.0.0.1:64211

Hi!

127.0.0.1:64208: Hi!

127.0.0.1:64208: Hi!

Hi yourself.

127.0.0.1:64211: Hi yourself.

127.0.0.1:64211: Hi yourself.

^C

127.0.0.1:64208 has left

$ ./netcat3

You are 127.0.0.1:64216

127.0.0.1:64216 has arrived

Welcome.

127.0.0.1:64211: Welcome.

127.0.0.1:64211: Welcome.

^C

127.0.0.1:64211 has left

While hosting a chat session for n clients, this program runs 2n+2 conc urrently communic ating goroutines, yet it needs no explicit locking operat ions (§9.2). The clients map is confine d to a single goroutine, the bro adc aster, so it cannot be accessed con cur rently. The only var iables that are share d by multiple goroutines are channel s and ins tances of net.Conn, bot h of which are concur rency saf e. We’ll tal k more about confinement, con cur rency safet y, and the implic ations of sharing var iables across goroutines in the next chapt er.

ptg16091132

Exercis e 8.12: Make the bro adc aster announce the cur rent set of clients to each new arrival.

This requires that the clients set and the entering and leaving ch annel s re cord the client name too.

Exercis e 8.13: Make the chat ser ver disconne ct idle clients, such as those that have sent no mess ages in the last five minut es. Hint: cal ling conn.Close() in anot her goroutine unb locks ac tive Read calls such as the one don e by input.Scan().

Exercis e 8.14: Change the chat ser ver’s net wor k prot ocol so that each client provides its name on ent ering . Us e that name ins tead of the networ k address when prefixing each message wit h its sender’s identity.

Exercis e 8.15: Failure of any client program to read dat a in a timely manner ult imate ly causes al l clients to get stuck. Modif y the bro adc aster to skip a message rat her than wait if a client wr iter is not ready to accept it. Alt ernat ive ly, add buf fer ing to each client’s out going message ch annel so that most messages are not dro p ped; the bro adc aster should use a non-blo cking send to this channel.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

9

Concurrency with

Shared Variables

In the pre vious chapt er, we present ed several programs that use goroutines and channel s to express con cur rency in a direc t and natural way. How ever, in doing so, we glossed over a ptg16091132

numb er of imp ortant and subtle issues that programmers must bear in mind when writing conc urrent code.

In this chapt er, we’ll take a clos er lo ok at the mech anics of con cur rency. In par tic ular, we’ll point out som e of the pro blems ass oci ated wit h sh aring var iables amon g mu ltiple goroutines, the analytical techniques for recog nizing those pro blems, and the patterns for solv ing them.

Final ly, we’ll explain som e of the technic al dif ferences bet ween goroutines and operat ing system threads.

9.1. Race Conditions

In a sequential program, that is, a program wit h on ly one goroutine, the steps of the program happ en in the fami liar exec ution order deter mined by the program logic. For ins tance, in a sequence of statements, the firs t on e happ ens before the secon d on e, and so on. In a program with two or more goroutines, the steps wit hin each goroutine happen in the fami liar order, but in general we don’t know whether an event x in one goroutine happens before an event y in anot her goroutine, or happens after it, or is simultane ous wit h it. When we cannot confidently say that one event happ ens before the other, then the events x and y are concur rent.

Consider a function that wor ks cor rec tly in a sequential program. That function is concurrenc y-s afe if it continues to wor k correc tly even when cal le d conc urrently, that is, fro m two or more goroutines wit h no addition al synchro nizat ion. We can generalize this not ion to a set of 257

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

258

CHAPTER 9. CONCURRENCY WITH SHARED VARIABLES

collab orat ing functions, such as the methods and operat ions of a par tic ular typ e. A type is conc urrency-s afe if all its accessible met hods and operat ions are con cur rency-s afe.

We can make a program conc urrency-s afe wit hout mak ing every con crete typ e in that prog ram conc urrency-s afe. Indeed, con cur rency-s afe typ es are the exception rat her than the rule, so you should access a var iable conc urrently only if the documentation for its typ e says that this is safe. We avoid conc urrent access to most var iables either by confini n g them to a single goroutine or by maintaining a hig her-le vel invar iant of mu tual exclu sion. We’ll explain thes e terms in this chapt er.

In contrast, expor ted package-le vel functions are general ly exp ected to be con cur rency-s afe.

Since package-le vel var iables cannot be confine d to a single goroutine, functions that modif y them must enforce mut ual exclusion.

There are many reasons a function mig ht not wor k when cal le d conc urrently, including deadlo ck, live lock, and res ource starvat ion. We don’t have space to dis cuss all of them, so we’ll fo cus on the most imp ortant one, the ra c e condit ion.

A race condit ion is a sit uat ion in which the program does not give the cor rec t resu lt for som e interleavings of the operat ions of multiple goroutines. Race condit ion s are per nicious because they may remain latent in a program and app ear infre quently, perhaps only under heavy load or when using cer tain comp ilers, platfor ms, or arc hit ectures. This makes them hard to reproduce and diagnos e.

ptg16091132

It is tradition al to explain the ser iousness of race condit ion s thro ugh the metaph or of financial loss, so we’ll con sider a simple ban k account program.

// Package bank implements a bank with only one account.

package bank

var balance int

func Deposit(amount int) { balance = balance + amount }

func Balance() int { return balance }

(We cou ld have writt en the body of the Deposit func tion as balance += amount, which is equivalent, but the lon g er for m wi l l simplif y the explanation.) Fo r a program this trivial, we can see at a glance that any sequence of cal ls to Deposit and Balance wi l l give the rig ht answer, that is, Balance wi l l report the sum of all amounts previously dep osite d. Ho wever, if we cal l thes e func tions not in a sequence but con cur rently, Balance is no lon g er guarante e d to give the rig ht answer. Con sider the fol low ing two goro utines, which represent two transactions on a joint ban k account:

// Alice:

go func() {

bank.Deposit(200)

// A1

fmt.Println("=", bank.Balance()) // A2

}()

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 9.1. RACE CONDITIONS

259

// Bob:

go bank.Deposit(100)

// B

Alice deposits $200, then che cks her bal ance, whi le Bob dep osits $100. Since the steps A1 and A2 occur con cur rently wit h B, we cannot pre dic t the order in which the y happ en. Int uit ive ly, it mig ht seem that there are only three possible order ings, which we’ll cal l ‘‘Alice firs t,’’ ‘‘Bob firs t,’’ and ‘‘Alice/B ob/A lice.’’ The fol low ing table shows the value of the balance var iable after each step. The quote d st rings represent the print ed bal ance slips.

Alice first

Bob first

Alice/Bob/Alice

0

0

0

A1

200

B

100

A1

200

A2 "= 200"

A1

300

B

300

B

300

A2 "= 300"

A2 "= 300"

In all cas es the final bal ance is $300. The only var iat ion is whether Alice’s bal ance slip includes Bob’s transaction or not, but the customers are sat isfied eit her way.

But this int uit ion is wro ng. There is a fourth possible out com e, in which Bob’s dep osit occ urs in the midd le of Alice’s dep osit, after the bal ance has been read (balance + amount) but before it has been updated (balance = ...), causing Bob’s transaction to dis app ear. This is because Alice’s dep osit operat ion A1 is really a sequence of two operat ions, a read and a write; cal l them A1r and A1w. Here’s the pro blemat ic interleaving:

Data race

ptg16091132

0

A1r

0

... = balance + amount

B

100

A1w

200

balance = ...

A2 "= 200"

Af ter A1r, the expression balance + amount evaluates to 200, so this is the value writt en during A1w, despit e the int ervening dep osit. The final bal ance is only $200. The ban k is $100

rich er at Bob’s exp ense.

This program cont ains a par tic ular kind of race condit ion cal le d a data race. A data race occurs whene ver two goro utines access the same var iable conc urrently and at least one of the accesses is a write.

Things get even messier if the dat a race invo l ves a var iable of a typ e that is larger than a single machine word, such as an int erface, a str ing , or a slice. This code updates x conc urrently to two slices of dif ferent lengt hs:

var x []int

go func() { x = make([]int, 10) }()

go func() { x = make([]int, 1000000) }()

x[999999] = 1 // NOTE: undefined behavior; memory corruption possible!

The value of x in the final statement is not define d; it cou ld be nil, or a slice of lengt h 10, or a slice of lengt h 1,000,000. But recal l that there are three parts to a slice: the point er, the lengt h, and the cap acity. If the point er com es fro m the firs t call to make and the lengt h comes fro m

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

260

CHAPTER 9. CONCURRENCY WITH SHARED VARIABLES

the secon d, x would be a chimera, a slice whose nominal lengt h is 1,000,000 but whose underly ing array has only 10 elements. In this event uality, storing to element 999,999 wou ld clobb er an arbit rar y faraway memor y lo cat ion, wit h cons equences that are imp ossible to pre dic t and hard to debug and localize. This semantic minefield is cal le d un define d behavior and is wel l know n to C programmers; for tunately it is rarely as tro ubles ome in Go as in C.

Ev en the notion that a con cur rent program is an int erleaving of several sequential programs is a false int uit ion. As we’ll see in Sec tion 9.4, dat a races may have even stranger out com es.

Many programmers—even som e very cle ver ones—wi l l occasionally offer justification s for know n data races in their programs: ‘‘the cost of mut ual exclusion is too hig h,’’ ‘‘this log ic is on ly for log ging,’’ ‘‘I don’t mind if I dro p some mess ages,’’ and so on. The abs ence of pro blems on a given comp iler and platfor m may give them false confidence. A go o d rule of thumb is that th ere is no such thi n g as a benig n data race. So how do we avoid dat a races in our prog rams?

We’ll rep eat the definition, since it is so imp ortant: A dat a race occ urs whene ver two goro utines access the same var iable conc urrently and at least one of the accesses is a write. It fo llows fro m this definition that there are three ways to avoid a dat a race.

The firs t way is not to write the var iable. Con sider the map below, which is lazi ly popu lated as each key is requeste d for the firs t time. If Icon is cal le d sequential ly, the program wor ks fine, but if Icon is cal le d conc urrently, there is a dat a race accessing the map.

ptg16091132

var icons = make(map[string]image.Image)

func loadIcon(name string) image.Image

// NOTE: not concurrency-safe!

func Icon(name string) image.Image {

icon, ok := icons[name]

if !ok {

icon = loadIcon(name)

icons[name] = icon

}

return icon

}

If ins tead we initialize the map wit h al l ne cessary ent ries before cre ating addition al goroutines and never modif y it again, then any number of goroutines may safely cal l Icon conc urrently since each only reads the map.

var icons = map[string]image.Image{

"spades.png":

loadIcon("spades.png"),

"hearts.png":

loadIcon("hearts.png"),

"diamonds.png": loadIcon("diamonds.png"),

"clubs.png":

loadIcon("clubs.png"),

}

// Concurrency-safe.

func Icon(name string) image.Image { return icons[name] }

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 9.1. RACE CONDITIONS

261

In the example above , the icons var iable is assig ned dur ing package initializat ion, which happ ens before the program’s main func tion starts running. Once initialize d, icons is never mo dified. Dat a st ruc tures that are never modified or are immut able are inherently con currenc y-s afe and need no synchro nizat ion. But obv iou sly we can’t use this appro ach if updates are ess ent ial, as wit h a ban k account.

The secon d way to avoid a dat a race is to avoid accessing the var iable fro m mu ltiple goro utines. This is the appro ach taken by many of the programs in the pre vious chapt er. For example, the main goroutine in the con cur rent web craw ler (§8.6) is the sole goroutine that accesses the seen map, and the broadcaster goro utine in the chat ser ver (§8.10) is the only goro utine that accesses the clients map. These var iables are confine d to a single goroutine.

Since other goroutines cannot access the var iable direc tly, the y mu st use a channel to send the confining goroutine a request to quer y or update the var iable. This is what is meant by the Go mant ra ‘‘Do not communic ate by sharing memor y; ins tead, share memor y by communic ating.’’ A goroutine that bro kers access to a confine d var iable using channel requests is cal le d a moni tor goroutine for that var iable. For example, the broadcaster goro utine monitors access to the clients map.

Here’s the ban k example rewritt en wit h the balance var iable confine d to a monit or goro utine called teller:

gopl.io/ch9/bank1

ptg16091132

// Package bank provides a concurrency-safe bank with one account.

package bank

var deposits = make(chan int) // send amount to deposit var balances = make(chan int) // receive balance

func Deposit(amount int) { deposits <- amount }

func Balance() int

{ return <-balances }

func teller() {

var balance int // balance is confined to teller goroutine for {

select {

case amount := <-deposits:

balance += amount

case balances <- balance:

}

}

}

func init() {

go teller() // start the monitor goroutine

}

Ev en when a var iable cannot be confine d to a single goroutine for its ent ire lifet ime, confinement may still be a solut ion to the pro blem of con cur rent access. For example, it’s common to sh are a var iable bet ween goroutines in a pip eline by passing its address fro m on e st age to the next over a channel. If each stage of the pip eline ref rains fro m accessing the var iable after

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

262

CHAPTER 9. CONCURRENCY WITH SHARED VARIABLES

sending it to the next stage , then all accesses to the var iable are sequential. In effec t, the var iable is confine d to one stage of the pip eline, then confine d to the next, and so on. This dis ci-pline is som etimes cal le d serial confinement.

In the example below, Cakes are ser ial ly confine d, firs t to the baker goro utine, then to the icer goro utine:

type Cake struct{ state string }

func baker(cooked chan<- *Cake) {

for {

cake := new(Cake)

cake.state = "cooked"

cooked <- cake // baker never touches this cake again

}

}

func icer(iced chan<- *Cake, cooked <-chan *Cake) {

for cake := range cooked {

cake.state = "iced"

iced <- cake // icer never touches this cake again

}

}

ptg16091132

The third way to avoid a dat a race is to allow many goroutines to access the var iable, but only on e at a time. This appro ach is known as mu tual exclu sion and is the subjec t of the next section.

Exercis e 9.1: Add a function Withdraw(amount int) bool to the gopl.io/ch9/bank1

prog ram. The result should indic ate whether the transaction succe e de d or fai le d due to ins ufficient funds. The mess age sent to the monitor goroutine must contain bot h the amount to withdraw and a new channel over which the monitor goroutine can send the boole an resu lt back to Withdraw.

9.2. Mutual Exclusion: sync.Mutex

In Sec tion 8.6, we used a buf fered channel as a counting semap hore to ens ure that no more than 20 goroutines made simultane ous HTTP requests. Wit h the same ide a, we can use a ch annel of cap acity 1 to ens ure that at most one goroutine accesses a share d var iable at a time.

A semaphore that counts only to 1 is cal le d a binary semap hore.

gopl.io/ch9/bank2

var (

sema

= make(chan struct{}, 1) // a binary semaphore guarding balance balance int

)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 9.2. MUTUAL EXCLUSION: SYNC.MUTEX

263

func Deposit(amount int) {

sema <- struct{}{} // acquire token

balance = balance + amount

<-sema // release token

}

func Balance() int {

sema <- struct{}{} // acquire token

b := balance

<-sema // release token

return b

}

This pattern of mu tual exclu sion is so useful that it is sup por ted direc tly by the Mutex type from the sync package. Its Lock method acquires the token (cal le d a lock) and its Unlock method releases it:

gopl.io/ch9/bank3

import "sync"

var (

mu

sync.Mutex // guards balance

balance int

)

func Deposit(amount int) {

ptg16091132

mu.Lock()

balance = balance + amount

mu.Unlock()

}

func Balance() int {

mu.Lock()

b := balance

mu.Unlock()

return b

}

Each time a goroutine accesses the var iables of the ban k (just balance here), it must cal l the mu tex’s Lock method to acquire an exc lusive lock. If som e ot her goroutine has acquired the lo ck, this operat ion will blo ck unt i l the other goroutine cal ls Unlock and the lock becom es avai lable again. The mut ex guards the share d var iables. By convent ion, the var iables guarde d by a mut ex are declare d immediate ly after the declarat ion of the mut ex its elf. If you deviate from this, be sure to document it.

The reg ion of code bet ween Lock and Unlock in which a goroutine is fre e to read and modif y the share d var iables is cal le d a cr iti cal sec tion. The lock holder’s cal l to Unlock happ ens before any other goroutine can acquire the lock for itself. It is ess ent ial that the goroutine release the lo ck once it is finishe d, on all paths through the function, including error pat hs.

The ban k prog ram ab ove exemp lifies a common con cur rency pattern. A set of exp orted function s encapsulates one or more var iables so that the only way to access the var iables is through

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

264

CHAPTER 9. CONCURRENCY WITH SHARED VARIABLES

thes e func tions (or met hods, for the var iables of an obj e ct). Each function acquires a mut ex lo ck at the beg inning and releases it at the end, there by ens uring that the shared var iables are not accessed con cur rently. This arrangement of functions, mut ex lock, and var iables is cal le d a moni tor. (This older use of the word ‘‘monitor’’ inspired the ter m ‘‘monitor goroutine.’’ Both us es share the meaning of a bro ker that ens ures var iables are accessed sequential ly.) Since the critical sec tion s in the Deposit and Balance func tions are so short—a single line, no branching—calling Unlock at the end is straig htfor ward. In more complex crit ical sec tion s, especi ally those in which error s mu st be dealt wit h by retur ning early, it can be hard to tel l that calls to Lock and Unlock are str ictly paired on all paths. Go’s defer st atement com es to the rescue: by defer r ing a cal l to Unlock, the critical sec tion imp licitly extends to the end of the current function, fre eing us fro m having to remember to ins ert Unlock calls in one or more pl aces far fro m the cal l to Lock.

func Balance() int {

mu.Lock()

defer mu.Unlock()

return balance

}

In the example above , the Unlock exec utes af ter the retur n st atement has read the value of balance, so the Balance func tion is con cur rency-s afe. As a bonus, we no lon g er need the lo cal variable b.

ptg16091132

Furthermore, a defer red Unlock wi l l run even if the critical sec tion panics, which may be important in programs that make use of recover (§5.10). A defer is marg inally more exp ensive than an explicit cal l to Unlock, but not enoug h to justify less cle ar co de. As always wit h conc urrent programs, favor clarity and resist premature opt imizat ion. Where possible, use defer and let crit ical sec tion s extend to the end of a function.

Consider the Withdraw func tion below. On success, it reduces the bal ance by the specified amount and retur ns true. But if the account holds ins ufficient funds for the transaction, Withdraw restores the bal ance and retur ns false.

// NOTE: not atomic!

func Withdraw(amount int) bool {

Deposit(-amount)

if Balance() < 0 {

Deposit(amount)

return false // insufficient funds

}

return true

}

This function event ual ly gives the cor rec t resu lt, but it has a nasty side effe ct. When an excessive wit hdrawal is att emp ted, the bal ance transiently dips below zero. This may cause a concurrent wit hdrawal for a modest sum to be spuriou sly rej e cte d. So if Bob tries to buy a sports car, Alice can’t pay for her mor ning cof fee. The pro blem is that Withdraw is not at o mic: it consists of a sequence of three sep arate operat ions, each of which acquires and then releases

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 9.2. MUTUAL EXCLUSION: SYNC.MUTEX

265

the mut ex lock, but not hing locks the whole sequence.

Ideally, Withdraw should acquire the mut ex lock once aro und the whole operat ion. How ever, this att emp t won’t wor k:

// NOTE: incorrect!

func Withdraw(amount int) bool {

mu.Lock()

defer mu.Unlock()

Deposit(-amount)

if Balance() < 0 {

Deposit(amount)

return false // insufficient funds

}

return true

}

Deposit tr ies to acquire the mut ex lock a secon d time by cal ling mu.Lock(), but because mu tex locks are not re-entrant—it’s not possible to lock a mut ex that’s already locke d—this le ads to a deadlock where not hing can pro ceed, and Withdraw blocks forever.

There is a good reason Go’s mut exes are not re-entrant. The pur pos e of a mut ex is to ens ure that certain invar iants of the shared var iables are maintained at critical points dur ing program ptg16091132

exec ution. One of the invar iants is ‘‘no goroutine is accessing the shared var iables,’’ but there may be addition al invar iants specific to the dat a st ruc tures that the mut ex guard s. When a goro utine acquires a mut ex lock, it may assume that the invar iants hold. Whi le it holds the lo ck, it may update the shared var iables so that the invar iants are temporar ily violate d.

Ho wever, when it releases the lock, it must guarante e that order has been restore d and the invar iants hold once again. Alt hough a re-entrant mut ex wou ld ens ure that no other goro utines are accessing the shared var iables, it cannot pro tec t the addition al invar iants of thos e var iables.

A common solut ion is to div ide a function such as Deposit into two: an unexp orted function, deposit, that assumes the lock is already held and does the real wor k, and an exp orted function Deposit that acquires the lock before cal ling deposit. We can then express Withdraw in terms of deposit li ke this:

func Withdraw(amount int) bool {

mu.Lock()

defer mu.Unlock()

deposit(-amount)

if balance < 0 {

deposit(amount)

return false // insufficient funds

}

return true

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

266

CHAPTER 9. CONCURRENCY WITH SHARED VARIABLES

func Deposit(amount int) {

mu.Lock()

defer mu.Unlock()

deposit(amount)

}

func Balance() int {

mu.Lock()

defer mu.Unlock()

return balance

}

// This function requires that the lock be held.

func deposit(amount int) { balance += amount }

Of course, the deposit func tion shown here is so trivial that a realist ic Withdraw func tion wouldn’t bot her cal ling it, but non etheless it illustrates the princip le.

Encapsulat ion (§6.6), by reducing unexp ected int erac tions in a program, helps us maintain data str ucture invar iants. For the same reason, encapsulat ion als o helps us maintain conc urrenc y invar iants. When you use a mut ex, make sure that bot h it and the var iables it guard s are not exp orted, whether the y are package-le vel var iables or the fields of a str uct.

9.3. Read/Write Mutexes: sync.RWMutex

ptg16091132

In a fit of anxiet y af ter seeing his $100 deposit vanish wit hout a trace, Bob writes a program to ch eck his ban k balance hundreds of times a secon d. He runs it at hom e, at wor k, and on his ph one. The ban k notices that the incre ased traffic is del aying dep osits and wit hdrawals, becaus e al l the Balance re quests run sequential ly, holding the lock exc lusively and temporarily pre venting other goroutines fro m running.

Since the Balance func tion only needs to re ad the state of the var iable, it wou ld in fac t be safe for multiple Balance calls to run conc urrently, so lon g as no Deposit or Withdraw call is running. In this scenar io we need a speci al kind of lock that allows read-only operat ions to proceed in paral lel wit h each other, but write operat ions to have fully exc lusive access. This lo ck is cal le d a mu ltipl e re aders, single writer lo ck, and in Go it’s provide d by sync.RWMutex: var mu sync.RWMutex

var balance int

func Balance() int {

mu.RLock() // readers lock

defer mu.RUnlock()

return balance

}

The Balance func tion now cal ls the RLock and RUnlock methods to acquire and release a re aders or sh are d lo ck. The Deposit func tion, which is unchange d, cal ls the mu.Lock and mu.Unlock methods to acquire and release a writ er or ex clu siv e lo ck.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 9.4. MEMORY SYNCHRONIZATION

267

Af ter this change , most of Bob’s Balance re quests run in paral lel wit h each other and finish more quickly. The lock is avai lable for more of the time, and Deposit re quests can pro ceed in a timely manner.

RLock can be used only if there are no writes to share d var iables in the critical sec tion. In general, we should not assume that logically re ad-only functions or met hods don’t als o up date some var iables. For example, a met hod that app ears to be a simple accessor mig ht als o increment an int ernal usage count er, or update a cache so that rep eat cal ls are faster. If in dou bt, us e an exclusive Lock.

It’s only profitable to use an RWMutex when most of the goroutines that acquire the lock are re aders, and the lock is under cont ent ion, that is, goroutines routinely have to wait to acquire it. An RWMutex re quires more complex internal bookkeeping , making it slower than a regu lar mu tex for uncontended locks.

9.4. Memory Synchronization

Yo u may won der why the Balance method needs mut ual exclusion, either channel-b ased or mu tex-b ased. After all, unlike Deposit, it con sists only of a single operat ion, so there is no danger of another goroutine exe cut ing ‘‘in the midd le’’ of it. There are two reasons we need a mu tex. The firs t is that it’s equ ally imp ortant that Balance not exe cut e in the midd le of som e ptg16091132

ot her operat ion like Withdraw. The secon d (and more subtle) reason is that synchro nizat ion is about more than just the order of exe cut ion of multiple goroutines; synchronizat ion als o af fec ts memor y.

In a moder n comp uter there may be dozens of pro cessors, each wit h its own local cache of the main memor y. For efficiency, writes to memor y are buf fered wit hin each pro cessor and flushe d out to main memor y on ly when necessary. The y may even be committ ed to main memory in a dif ferent order than the y were writt en by the writing goro utine. Synchro nizat ion pr imitives like channel communic ations and mut ex operat ions cause the pro cessor to flush out and commit all its accumulated writes so that the effec ts of goroutine exe cut ion up to that point are guarante e d to be visible to goroutines running on other pro cessors.

Consider the possible out puts of the fol low ing snipp et of code: var x, y int

go func() {

x = 1

// A1

fmt.Print("y:", y, " ") // A2

}()

go func() {

y = 1

// B1

fmt.Print("x:", x, " ") // B2

}()

Since these two goro utines are con cur rent and access share d var iables wit hout mut ual exclusion, there is a dat a race, so we should not be sur prised that the program is not

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

268

CHAPTER 9. CONCURRENCY WITH SHARED VARIABLES

deter minist ic. We mig ht exp ect it to print any one of these four results, which cor respond to intuit ive int erleavings of the lab ele d st atements of the program: y:0 x:1

x:0 y:1

x:1 y:1

y:1 x:1

The fourth line cou ld be explained by the sequence A1,B1,A2,B2 or by B1,A1,A2,B2, for example. How ever, these two out com es mig ht com e as a sur prise: x:0 y:0

y:0 x:0

but dep ending on the compi ler, CPU, and many other factors, the y can happen too. What possible interleaving of the four statements cou ld explain them?

Wi thin a single goroutine, the effec ts of each statement are guarante e d to occ ur in the order of exec ution; goroutines are sequ ent ial ly consi stent. But in the abs ence of explicit synchro nization using a channel or mut ex, there is no guarante e that events are seen in the same order by al l goro utines. Alt hough goroutine A mu st obs erve the effec t of the write x = 1 before it reads the value of y, it does not necessarily obs erve the write to y done by goroutine B, so A may pr int a stale value of y.

ptg16091132

It is tempt ing to try to underst and con cur rency as if it cor respond s to some interleaving of the st atements of each goroutine, but as the example above shows, this is not how a moder n compi ler or CPU wor ks. Because the assig nment and the Print refer to dif ferent var iables, a compi ler may conclude that the order of the two statements cannot affec t the result, and swap them. If the two goro utines exe cut e on dif ferent CPUs, each wit h its own cach e, wr ites by one goro utine are not visible to the other goroutine’s Print until the caches are synchro nize d with main memor y.

Al l thes e conc urrency pro blems can be avoide d by the con sistent use of simple, est ablishe d patt erns. Where possible, confine var iables to a single goroutine; for all other variables, use mu tual exc lusion.

9.5. Lazy Initialization: sync.Once

It is good prac tice to defer an expensive initializat ion step unt i l the mom ent it is needed. Initializing a var iable up fro nt incre ases the start-up latency of a program and is unnecessary if exec ution doesn’t always reach the par t of the program that uses that var iable. Let’s retur n to the icons var iable we saw earlier in the chapt er:

var icons map[string]image.Image

This version of Icon us es lazy ini tializati on:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 9.5. LAZY INITIALIZATION: SYNC.ONCE

269

func loadIcons() {

icons = map[string]image.Image{

"spades.png":

loadIcon("spades.png"),

"hearts.png":

loadIcon("hearts.png"),

"diamonds.png": loadIcon("diamonds.png"),

"clubs.png":

loadIcon("clubs.png"),

}

}

// NOTE: not concurrency-safe!

func Icon(name string) image.Image {

if icons == nil {

loadIcons() // one-time initialization

}

return icons[name]

}

Fo r a var iable accessed by only a single goroutine, we can use the pattern above , but this pattern is not safe if Icon is cal le d conc urrently. Like the ban k’s original Deposit func tion, Icon consists of multiple steps: it tests whether icons is nil, then it loads the icons, then it updates icons to a non-ni l value. Int uit ion mig ht sug gest that the wor st possible out com e of the race condit ion above is that the loadIcons func tion is cal le d several times. Whi le the firs t goro utine is busy loading the icons, another goroutine ent ering Icon would find the var iable ptg16091132

st i l l equal to nil, and wou ld als o call loadIcons.

But this int uit ion is als o wron g. (We hop e that by now you are developing a new int uit ion ab out con cur rency, that int uit ion s ab out con cur rency are not to be trusted!) Recal l the discussion of memor y from Sec tion 9.4. In the abs ence of explicit synchro nizat ion, the compi ler and CPU are fre e to reorder accesses to memor y in any number of ways, so lon g as the beh avior of each goroutine is sequential ly con sistent. One possible reorder ing of the statements of loadIcons is shown below. It stores the emp ty map in the icons var iable before popu lat ing it: func loadIcons() {

icons = make(map[string]image.Image)

icons["spades.png"] = loadIcon("spades.png") icons["hearts.png"] = loadIcon("hearts.png") icons["diamonds.png"] = loadIcon("diamonds.png") icons["clubs.png"] = loadIcon("clubs.png")

}

Cons equently, a goroutine finding icons to be non-ni l may not assume that the initializat ion of the var iable is complete.

The simplest cor rec t way to ens ure that all goroutines obs erve the effec ts of loadIcons is to sy nchro nize them using a mut ex:

var mu sync.Mutex // guards icons

var icons map[string]image.Image

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

270

CHAPTER 9. CONCURRENCY WITH SHARED VARIABLES

// Concurrency-safe.

func Icon(name string) image.Image {

mu.Lock()

defer mu.Unlock()

if icons == nil {

loadIcons()

}

return icons[name]

}

Ho wever, the cost of enforcing mut ual ly exc lusive access to icons is that two goro utines cannot access the var iable conc urrently, even once the var iable has been safely initialize d and will ne ver be modified again. This sug gests a multiple-readers lock: var mu sync.RWMutex // guards icons

var icons map[string]image.Image

// Concurrency-safe.

func Icon(name string) image.Image {

mu.RLock()

if icons != nil {

icon := icons[name]

mu.RUnlock()

return icon

ptg16091132

}

mu.RUnlock()

// acquire an exclusive lock

mu.Lock()

if icons == nil { // NOTE: must recheck for nil

loadIcons()

}

icon := icons[name]

mu.Unlock()

return icon

}

There are now two critical sec tion s. The goroutine firs t acquires a reader lock, con sults the map, then releases the lock. If an ent ry was found (the common cas e), it is retur ned. If no entr y was found, the goroutine acquires a writer lock. There is no way to upg rade a share d lo ck to an exc lusive one wit hout firs t re leasing the shared lock, so we must reche ck the icons var iable in cas e anot her goroutine already initialize d it in the int erim.

The pattern above gives us gre ater con cur rency but is complex and thu s er ror-pro ne.

Fo rtunate ly, the sync package provides a speci alized solut ion to the pro blem of one-t ime initializat ion: sync.Once. Con ceptu ally, a Once consists of a mut ex and a boole an var iable that re cords whether initializat ion has taken place; the mut ex guard s both the boole an and the client’s dat a st ruc tures. The sole met hod, Do, accepts the initializat ion function as its argument. Let’s use Once to simplif y the Icon func tion:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 9.6. THE RACE DETECTOR

271

var loadIconsOnce sync.Once

var icons map[string]image.Image

// Concurrency-safe.

func Icon(name string) image.Image {

loadIconsOnce.Do(loadIcons)

return icons[name]

}

Each cal l to Do(loadIcons) lo cks the mut ex and che cks the boole an var iable. In the firs t call, in which the var iable is false, Do calls loadIcons and sets the var iable to true. Subsequent calls do not hing, but the mut ex synchro nizat ion ens ures that the effec ts of loadIcons on memory (sp ecifically, icons) becom e visible to all goroutines. Using sync.Once in this way, we can avoid sharing var iables wit h ot her goroutines unt i l they have been pro perly const ruc ted.

Exercis e 9.2: Re writ e the PopCount example fro m Section 2.6.2 so that it initializes the looku p table using sync.Once the firs t time it is needed. (Re alist ically, the cost of synchro nizat ion would be pro hibit ive for a small and hig hly opt imize d func tion like PopCount.) 9.6. The Race Det ector

ptg16091132

Ev en wit h the gre atest of care, it’s all too easy to make con cur rency mistakes. For tunately, the Go runtime and toolchain are equip ped wit h a sop histicate d and easy-to-use dynamic analysis to ol, the ra c e dete ctor.

Ju st add the -race flag to your go build, go run, or go test command. This causes the compi ler to bui ld a modified version of your app lic ation or test wit h addition al ins trument ation that effe ctive ly records all accesses to share d var iables that occ urre d during exe cut ion, along with the identity of the goroutine that read or wro te the var iable. In addition, the modified prog ram re cords all synchro nizat ion events, such as go st atements, channel operat ions, and calls to (*sync.Mutex).Lock, (*sync.WaitGroup).Wait, and so on. (The complete set of sy nchro nizat ion events is specified by the The Go Memor y Mo d el do cument that accomp anies the langu age specification.)

The race detec tor studies this stream of events, look ing for cas es in which one goroutine reads or writes a share d var iable that was most recently writt en by a dif ferent goroutine wit hout an intervening synchro nizat ion operat ion. This indic ates a conc urrent access to the shared var iable, and thu s a dat a race. The tool prints a rep ort that includes the identity of the var iable, and the stacks of active function cal ls in the reading goroutine and the writing goro utine. This is usually sufficient to pinpoint the pro blem. Sec tion 9.7 cont ains an example of the race detec tor in action.

The race detec tor rep orts all dat a races that were actually exe cut ed. How ever, it can only detec t race condit ion s that occ ur during a run; it cannot prove that non e wi l l ever occ ur. For best results, make sure that your tests exercise your packages using con cur rency.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

272

CHAPTER 9. CONCURRENCY WITH SHARED VARIABLES

Due to ext ra bookkeeping , a program bui lt wit h race detec tion needs more time and memory to run, but the overhe ad is tolerable even for many pro duc tion jobs. For inf requently occ urring race condit ion s, letting the race detec tor do its job can save hours or days of debug ging.

9.7. Example: Concurrent Non-Blocking Cache

In this sec tion, we’ll bui ld a concur rent non-blocki n g cache, an abstrac tion that solves a problem that arises often in real-wor ld con cur rent programs but is not wel l addressed by exist ing librar ies. This is the pro blem of memoizing a function, that is, caching the result of a func tion so that it need be compute d on ly once. Our solut ion will be con cur rency-s afe and wi l l avoid the content ion ass oci ated wit h desig ns bas ed on a single lock for the whole cache.

We’ll use the httpGetBody func tion below as an example of the typ e of function we mig ht want to memoize. It makes an HTTP GET request and reads the request body. Cal ls to this func tion are relative ly exp ensive , so we’d like to avoid rep eat ing them unnecessarily.

func httpGetBody(url string) (interface{}, error) {

resp, err := http.Get(url)

if err != nil {

return nil, err

}

ptg16091132

defer resp.Body.Close()

return ioutil.ReadAll(resp.Body)

}

The final line hides a minor subtlet y. ReadAll returns two results, a []byte and an error, but since these are assig nable to the declare d resu lt typ es of httpGetBody—interface{} and error, respectively—we can retur n the result of the cal l without fur ther ado. We chose this return typ e for httpGetBody so that it conforms to the typ e of functions that our cache is desig ned to memoize.

Here’s the firs t draf t of the cache:

gopl.io/ch9/memo1

// Package memo provides a concurrency-unsafe

// memoization of a function of type Func.

package memo

// A Memo caches the results of calling a Func.

type Memo struct {

f

Func

cache map[string]result

}

// Func is the type of the function to memoize.

type Func func(key string) (interface{}, error)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 9.7. EXAMPLE: CONCURRENT NON-BLOCKING CACHE

273

type result struct {

value interface{}

err

error

}

func New(f Func) *Memo {

return &Memo{f: f, cache: make(map[string]result)}

}

// NOTE: not concurrency-safe!

func (memo *Memo) Get(key string) (interface{}, error) {

res, ok := memo.cache[key]

if !ok {

res.value, res.err = memo.f(key)

memo.cache[key] = res

}

return res.value, res.err

}

A Memo inst ance holds the function f to memoize, of typ e Func, and the cache, which is a mapping fro m st rings to results. Each result is simply the pair of results retur ned by a cal l to f—a value and an error. We’ll show several variation s of Memo as the desig n prog resses, but al l wi l l sh are these basic aspects.

ptg16091132

An example of how to use Memo appears below. For each element in a stream of incoming URLs, we cal l Get, log ging the latency of the cal l and the amount of dat a it retur ns: m := memo.New(httpGetBody)

for url := range incomingURLs() {

start := time.Now()

value, err := m.Get(url)

if err != nil {

log.Print(err)

}

fmt.Printf("%s, %s, %d bytes\n",

url, time.Since(start), len(value.([]byte)))

}

We can use the testing package (the topic of Chapt er 11) to systematically inv est igate the ef fec t of memoizat ion. Fro m the test out put below, we see that the URL stream contains dup lic ates, and that alt hough the firs t call to (*Memo).Get for each URL takes hundreds of mi l lisecond s, the secon d re quest retur ns the same amount of dat a in under a millis econd.

$ go test -v gopl.io/ch9/memo1

=== RUN

Test

https://golang.org, 175.026418ms, 7537 bytes

https://godoc.org, 172.686825ms, 6878 bytes

https://play.golang.org, 115.762377ms, 5767 bytes

http://gopl.io, 749.887242ms, 2856 bytes

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

274

CHAPTER 9. CONCURRENCY WITH SHARED VARIABLES

https://golang.org, 721ns, 7537 bytes

https://godoc.org, 152ns, 6878 bytes

https://play.golang.org, 205ns, 5767 bytes

http://gopl.io, 326ns, 2856 bytes

--- PASS: Test (1.21s)

PASS

ok gopl.io/ch9/memo1

1.257s

This test exe cut es all cal ls to Get sequential ly.

Since HTTP requests are a gre at opp ortunity for paral lelism, let’s change the test so that it makes all requests conc urrently. The test uses a sync.WaitGroup to wait unt i l the last request is complete before retur ning.

m := memo.New(httpGetBody)

var n sync.WaitGroup

for url := range incomingURLs() {

n.Add(1)

go func(url string) {

start := time.Now()

value, err := m.Get(url)

if err != nil {

log.Print(err)

}

ptg16091132

fmt.Printf("%s, %s, %d bytes\n",

url, time.Since(start), len(value.([]byte)))

n.Done()

}(url)

}

n.Wait()

The test runs much faster, but unfor tunately it is unlikely to wor k correc tly all the time. We may not ice unexp ected cache misses, or cache hits that retur n incorrec t values, or even crashes.

Wo rse, it is likely to wor k correc tly some of the time, so we may not even not ice that it has a problem. But if we run it wit h the -race flag , the race detec tor (§9.6) often prints a rep ort such as this one:

$ go test -run=TestConcurrent -race -v gopl.io/ch9/memo1

=== RUN

TestConcurrent

...

WARNING: DATA RACE

Write by goroutine 36:

runtime.mapassign1()

~/go/src/runtime/hashmap.go:411 +0x0

gopl.io/ch9/memo1.(*Memo).Get()

~/gobook2/src/gopl.io/ch9/memo1/memo.go:32 +0x205

...

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 9.7. EXAMPLE: CONCURRENT NON-BLOCKING CACHE

275

Previous write by goroutine 35:

runtime.mapassign1()

~/go/src/runtime/hashmap.go:411 +0x0

gopl.io/ch9/memo1.(*Memo).Get()

~/gobook2/src/gopl.io/ch9/memo1/memo.go:32 +0x205

...

Found 1 data race(s)

FAIL

gopl.io/ch9/memo1

2.393s

The reference to memo.go:32 te lls us that two goro utines have updated the cache map without any int ervening synchro nizat ion. Get is not con cur rency-s afe: it has a dat a race.

28 func (memo *Memo) Get(key string) (interface{}, error) {

29

res, ok := memo.cache[key]

30

if !ok {

31

res.value, res.err = memo.f(key)

32

memo.cache[key] = res

33

}

34

return res.value, res.err

35 }

The simplest way to make the cache con cur rency-s afe is to use monit or-b ased synchro nization. All we need to do is add a mut ex to the Memo, acquire the mut ex lock at the start of Get, and release it before Get returns, so that the two cache op erat ions occ ur within the critical ptg16091132

section:

gopl.io/ch9/memo2

type Memo struct {

f

Func

mu

sync.Mutex // guards cache

cache map[string]result

}

// Get is concurrency-safe.

func (memo *Memo) Get(key string) (value interface{}, err error) {

memo.mu.Lock()

res, ok := memo.cache[key]

if !ok {

res.value, res.err = memo.f(key)

memo.cache[key] = res

}

memo.mu.Unlock()

return res.value, res.err

}

No w the race detec tor is silent, even when running the tests conc urrently. Unfor tunately this ch ange to Memo re verses our earlier per for mance gains. By holding the lock for the durat ion of each cal l to f, Get serializes all the I/O operat ions we int ended to paral lelize. What we need is a non-blocki n g cach e, on e that does not ser ialize cal ls to the function it memoizes.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

276

CHAPTER 9. CONCURRENCY WITH SHARED VARIABLES

In the next implementation of Get, below, the cal ling goroutine acquires the lock twice: once for the looku p, and then a secon d time for the update if the looku p returned not hing. In between, other goroutines are fre e to use the cache.

gopl.io/ch9/memo3

func (memo *Memo) Get(key string) (value interface{}, err error) {

memo.mu.Lock()

res, ok := memo.cache[key]

memo.mu.Unlock()

if !ok {

res.value, res.err = memo.f(key)

// Between the two critical sections, several goroutines

// may race to compute f(key) and update the map.

memo.mu.Lock()

memo.cache[key] = res

memo.mu.Unlock()

}

return res.value, res.err

}

The per for mance improv es again, but now we not ice that som e URLs are being fetch ed twice.

This happens when two or more goroutines cal l Get for the same URL at about the same time.

Both con sult the cache, find no value there , and then cal l the slow function f. Then bot h of ptg16091132

them update the map wit h the result the y obtained. One of the results is overwritt en by the ot her.

Ideally we’d like to avoid this redundant wor k. This feature is som etimes cal le d du p licate sup pressi on. In the version of Memo below, each map element is a point er to an entry st ruc t.

Each entry cont ains the memoize d resu lt of a cal l to the function f, as before, but it addition-al ly contains a channel cal le d ready. Just after the entry’s result has been set, this channel wi l l be clos ed, to broadcast (§8.9) to any other goroutines that it is now safe for them to read the result fro m the entry.

gopl.io/ch9/memo4

type entry struct {

res

result

ready chan struct{} // closed when res is ready

}

func New(f Func) *Memo {

return &Memo{f: f, cache: make(map[string]*entry)}

}

type Memo struct {

f

Func

mu

sync.Mutex // guards cache

cache map[string]*entry

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 9.7. EXAMPLE: CONCURRENT NON-BLOCKING CACHE

277

func (memo *Memo) Get(key string) (value interface{}, err error) {

memo.mu.Lock()

e := memo.cache[key]

if e == nil {

// This is the first request for this key.

// This goroutine becomes responsible for computing

// the value and broadcasting the ready condition.

e = &entry{ready: make(chan struct{})}

memo.cache[key] = e

memo.mu.Unlock()

e.res.value, e.res.err = memo.f(key)

close(e.ready) // broadcast ready condition

} else {

// This is a repeat request for this key.

memo.mu.Unlock()

<-e.ready // wait for ready condition

}

return e.res.value, e.res.err

}

A cal l to Get now inv olves acquir ing the mut ex lock that guard s the cache map, looking in the map for a point er to an exist ing entry, allocat ing and ins erting a new entry if non e was ptg16091132

found, then releasing the lock. If there was an exist ing entry, its value is not necessarily ready yet—anot her goroutine cou ld still be cal ling the slow function f—so the cal ling goroutine mu st wait for the entry’s ‘‘re ady’’ condit ion before it reads the entry’s result. It does this by re ading a value fro m the ready ch annel, since this operat ion blo cks until the channel is clos ed.

If there was no exist ing entry, then by ins erting a new ‘‘not ready’’ entry into the map, the current goroutine becom es responsible for inv oking the slow function, updat ing the entry, and bro adc asting the readiness of the new entry to any other goroutines that mig ht (by then) be waiting for it.

No tice that the var iables e.res.value and e.res.err in the entry are share d amon g mu ltiple goroutines. The goroutine that cre ates the entry sets their values, and other goro utines read their values once the ‘‘re ady’’ condit ion has been bro adc ast. Despit e being accessed by multiple goroutines, no mut ex lock is necessary. The closing of the ready ch annel happ ens before any other goroutine receives the bro adc ast event, so the write to those var iables in the firs t goro utine happ ens before they are read by subsequent goroutines. There is no dat a race.

Our con cur rent, dup lic ate-sup pressing, non-blo cking cache is complete.

The imp lementation of Memo ab ove uses a mut ex to guard a map variable that is share d by each goroutine that cal ls Get. It’s int erest ing to contrast this desig n with an alt ernat ive one in which the map variable is confine d to a moni tor goroutine to which cal lers of Get mu st send a mess age .

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

278

CHAPTER 9. CONCURRENCY WITH SHARED VARIABLES

The declarat ions of Func, result, and entry remain as before:

// Func is the type of the function to memoize.

type Func func(key string) (interface{}, error)

// A result is the result of calling a Func.

type result struct {

value interface{}

err

error

}

type entry struct {

res

result

ready chan struct{} // closed when res is ready

}

Ho wever, the Memo type now con sists of a channel, requests, through which the cal ler of Get communic ates wit h the monit or goro utine. The element typ e of the channel is a request.

Using this str ucture, the cal ler of Get send s the monit or goro utine bot h the key, that is, the argument to the memoize d func tion, and another channel, response, over which the result should be sent back when it becom es avai lable. This channel will car ry only a single value.

gopl.io/ch9/memo5

// A request is a message requesting that the Func be applied to key.

type request struct {

ptg16091132

key

string

response chan<- result // the client wants a single result

}

type Memo struct{ requests chan request }

// New returns a memoization of f.

Clients must subsequently call Close.

func New(f Func) *Memo {

memo := &Memo{requests: make(chan request)}

go memo.server(f)

return memo

}

func (memo *Memo) Get(key string) (interface{}, error) {

response := make(chan result)

memo.requests <- request{key, response}

res := <-response

return res.value, res.err

}

func (memo *Memo) Close() { close(memo.requests) }

The Get method, above , creates a respons e ch annel, puts it in the request, sends it to the monitor goroutine, then immediate ly receives fro m it.

The cache var iable is confine d to the monitor goroutine (*Memo).server, shown below. The monitor reads requests in a loop unt i l the request channel is clos ed by the Close method. For each request, it con sults the cache, cre ating and ins erting a new entry if non e was found.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 9.7. EXAMPLE: CONCURRENT NON-BLOCKING CACHE

279

func (memo *Memo) server(f Func) {

cache := make(map[string]*entry)

for req := range memo.requests {

e := cache[req.key]

if e == nil {

// This is the first request for this key.

e = &entry{ready: make(chan struct{})}

cache[req.key] = e

go e.call(f, req.key) // call f(key)

}

go e.deliver(req.response)

}

}

func (e *entry) call(f Func, key string) {

// Evaluate the function.

e.res.value, e.res.err = f(key)

// Broadcast the ready condition.

close(e.ready)

}

func (e *entry) deliver(response chan<- result) {

// Wait for the ready condition.

<-e.ready

// Send the result to the client.

ptg16091132

response <- e.res

}

In a simi lar manner to the mut ex-b ased version, the firs t re quest for a given key becom es resp onsible for cal ling the function f on that key, storing the result in the entry, and bro adcast ing the readiness of the entry by closing the ready ch annel. This is don e by (*entry).call.

A subsequent request for the same key find s the exist ing entry in the map, waits for the result to becom e re ady, and sends the result through the respons e ch annel to the client goroutine that cal le d Get. This is don e by (*entry).deliver. The call and deliver methods must be called in their own goroutines to ens ure that the monitor goroutine does not stop pro cessing ne w re quests.

This example shows that it’s possible to bui ld many con cur rent str uctures using eit her of the two appro ach es—sh are d var iables and locks, or communic ating sequential pro cesses—

without excessive complexity.

It’s not always obv iou s which appro ach is preferable in a given situ ation, but it’s wor th know ing how the y correspond. Som etimes switching fro m on e approach to the other can make your code simpler.

Exercis e 9.3: Extend the Func type and the (*Memo).Get method so that cal lers may provide an opt ion al done ch annel through which the y can cancel the operat ion (§8.9). The results of a cancel led Func call should not be cache d.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

280

CHAPTER 9. CONCURRENCY WITH SHARED VARIABLES

9.8. Goroutines and Threads

In the pre vious chapt er we said that the dif ference bet ween goroutines and operat ing system (OS) threads cou ld be ignored unt i l later. Alt hough the dif ferences bet ween them are ess ential ly quant itative , a big enough quant itative dif ference becom es a qualitative one, and so it is with goroutines and threads. The time has now com e to distinguish them.

9.8.1. Gro wable Stacks

Each OS thread has a fixe d-size blo ck of memor y (often as large as 2MB) for its sta ck, the wor k area where it saves the local variables of function cal ls that are in progress or temporar ily su spende d while another function is cal le d. This fixe d-size stack is simultane ously too much and too lit tle. A 2MB stack wou ld be a huge waste of memor y for a litt le goroutine, such as on e that merely waits for a WaitGroup then clos es a channel. It’s not uncommon for a Go prog ram to cre ate hundreds of thous and s of goroutines at one time, which wou ld be imp ossible wit h st acks this large . Yet despit e their size, fixe d-size stacks are not always big enough for the most complex and deeply rec ursive of functions. Chang ing the fixe d size can improv e sp ace efficiency and allow more threads to be cre ate d, or it can enable more deeply rec ursive func tions, but it cannot do bot h.

ptg16091132

In contrast, a goroutine starts life wit h a small stack, typic ally 2KB. A goro utine’s stack, like the stack of an OS thread, holds the local variables of active and suspended function cal ls, but un li ke an OS thread, a goroutine’s stack is not fixe d; it grows and shr inks as needed. The size limit for a goroutine stack may be as much as 1GB, orders of mag nitude larger than a typic al fixe d-size thread stack, thoug h of cours e fe w goro utines use that much.

Exercis e 9.4: Cons truct a pip eline that conne cts an arbit rar y numb er of goroutines wit h ch annel s. What is the maximum number of pip eline stages you can create wit hout running out of memory? How lon g do es a value take to transit the ent ire pip eline?

9.8.2. Goroutine Scheduling

OS threads are sch edu led by the OS ker nel. Every few millis econd s, a hardware timer inter-rupts the pro cessor, which causes a ker nel function cal le d the sche dul er to be inv oke d. This func tion suspend s the cur rently exe cut ing thread and saves its reg isters in memor y, looks over the list of threads and decides which one should run next, restores that thread’s reg isters fro m memory, then resumes the exe cut ion of that thread. Because OS threads are sch edu led by the kernel, passing control fro m on e thre ad to another requires a full cont ext switch, that is, sav ing the state of one user thread to memor y, restoring the state of another, and updat ing the sche duler’s dat a st ruc tures. This operat ion is slow, due to its poor locality and the number of memory accesses required, and has histor ically only got ten wor se as the number of CPU cyc les re quired to access memor y has incre ased.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 9.8. GOROUTINES AND THREADS

281

The Go runtime contains its own sch edu ler that uses a technique known as m:n sch edu ling, becaus e it multiplexes (or sch edu les) m goro utines on n OS threads. The job of the Go sche duler is analogou s to that of the ker nel sch edu ler, but it is con cer ned only wit h the goro utines of a single Go program.

Un like the operat ing system’s thread sch edu ler, the Go sch edu ler is not inv oke d periodic ally by a hardware timer, but imp licitly by cer tain Go langu age con str ucts. For example, when a goro utine cal ls time.Sleep or blo cks in a channel or mut ex operat ion, the sch edu ler puts it to sleep and runs another goroutine unt i l it is time to wake the firs t on e up. Because it doesn’t ne e d a switch to ker nel context, res che duling a goroutine is much che aper than res che duling a thre ad.

Exercis e 9.5: Wr ite a program wit h two goroutines that send messages back and for th over two unbuf fered channel s in ping-p ong fashion. How many communic ations per secon d can the program sustain?

9.8.3. GOMAXPROCS

The Go sch edu ler uses a parameter cal le d GOMAXPROCS to deter mine how many OS threads may be actively exe cut ing Go code simultane ously. Its default value is the number of CPUs on the machine, so on a machine wit h 8 CPUs, the sch edu ler will sch edu le Go code on up to 8 OS

ptg16091132

thre ads at once. (GOMAXPROCS is the n in m:n sche duling.) Goroutines that are sle eping or blocke d in a communic ation do not need a thread at all. Goroutines that are blo cke d in I/O or ot her system cal ls or are cal ling non-G o func tions, do need an OS thread, but GOMAXPROCS

ne e d not account for them.

Yo u can explicitly control this parameter using the GOMAXPROCS enviro nment var iable or the runtime.GOMAXPROCS func tion. We can see the effec t of GOMAXPROCS on this lit tle program, which prints an end less stream of zeros and ones:

for {

go fmt.Print(0)

fmt.Print(1)

}

$ GOMAXPROCS=1 go run hacker-cliché.go

111111111111111111110000000000000000000011111...

$ GOMAXPROCS=2 go run hacker-cliché.go

010101010101010101011001100101011010010100110...

In the firs t run, at most one goroutine was exec ute d at a time. Initial ly, it was the main goro utine, which prints ones. After a per iod of time, the Go sch edu ler put it to sle ep and woke up the goroutine that prints zeros, giv ing it a tur n to run on the OS thread. In the secon d run, there were two OS threads avai lable, so bot h goro utines ran simultane ously, print ing dig its at ab out the same rate. We must stress that many fac tor s are inv olved in goroutine sch edu ling, and the runtime is con stant ly evo l ving, so your results may differ fro m the ones above .

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

282

CHAPTER 9. CONCURRENCY WITH SHARED VARIABLES

Exercis e 9.6: Me asure how the per for mance of a compute-b ound paral lel program (see Exercise 8.5) varies wit h GOMAXPROCS. What is the opt imal value on your computer? How many CPUs does your computer have?

9.8.4. Goroutines Have No Identity

In most operat ing systems and programming langu ages that sup por t mu ltithre ading, the current thread has a dist inc t identity that can be easi ly obt ained as an ordinar y value, typic ally an integer or point er. This makes it easy to bui ld an abstrac tion cal le d thre ad-local storage, which is ess ent ial ly a global map keyed by thread identity, so that each thread can store and ret rie ve values indep endent of other threads.

Goro utines have no not ion of identity that is accessible to the programmer. This is by desig n, since thread-lo cal storage tends to be abu sed. For example, in a web ser ver implemente d in a language wit h thre ad-lo cal storage, it’s common for many functions to find infor mat ion about the HTTP request on whose beh alf the y are cur rently wor king by looking in that storage.

Ho wever, just as wit h prog rams that rely excessive ly on global var iables, this can lead to an un healt hy ‘‘ac tion at a distance’’ in which the beh avior of a function is not deter mined by its arguments alone, but by the identity of the thread in which it runs. Con sequently, if the identity of the thread should change—some wor ker threads are enliste d to help, say—the function misb ehaves mysteriou sly.

ptg16091132

Go encourages a simpler sty le of programming in which parameters that affec t the beh avior of a function are explicit. Not only does this make programs easier to read, but it lets us fre ely assig n su btasks of a given function to many dif ferent goroutines wit hout wor rying about their identity.

Yo u’ve now learned about all the langu age features you need for writing Go programs. In the next two chapt ers, we’ll step back to look at som e of the prac tices and tools that sup por t prog ramming in the large: how to str ucture a pro jec t as a set of packages, and how to obt ain, build, test, benchmark, profile, document, and share those packages.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

10

Packages and the Go Tool

A modest-size program today might contain 10,000 functions. Yet its author need thin k ab out on ly a few of them and desig n even fewer, because the vast maj ority were writt en by others and made available for reuse through pack ages.

Go com es wit h ov er 100 stand ard packages that provide the found ation s for most app lic ations.

ptg16091132

The Go community, a thr iving ecosystem of package desig n, sh aring , reus e, and imp rov ement, has publishe d many more, and you can find a searc hable index of them at http://godoc.org.

In this chapt er, we’ll show how to use exist ing packages and cre ate new ones.

Go als o comes wit h the go to ol, a sop histicate d but simple-to-use command for manag ing worksp aces of Go packages. Since the beg inning of the book, we’ve been showing how to use the go to ol to dow nlo ad, bui ld, and run example programs. In this chapt er, we’ll look at the to ol’s underly ing con cepts and tour more of its cap abilit ies, which include print ing documentation and quer ying met adat a ab out the packages in the wor ksp ace. In the next chapt er we’ll explore its testing features.

10.1. Introduction

The pur pos e of any package system is to make the desig n and maintenance of large programs prac tic al by gro uping rel ate d fe atures toget her into units that can be easi ly understood and ch ange d, indep endent of the other packages of the program. This mo dul arity al lows packages to be share d and reused by dif ferent pro jec ts, dist ribut ed wit hin an organizat ion, or made avai lable to the wider wor ld.

Each package defines a distinc t name space that enclos es its identifiers. Each name is ass ociated wit h a par tic ular package, letting us cho ose short, cle ar names for the typ es, func tions, and so on that we use most often, wit hout cre ating conflic ts with other par ts of the program.

283

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

284

CHAPTER 10. PACKAGES AND THE GO TOOL

Packages also provide encapsul ati on by control ling which names are visible or exp orted outside the package. Restr icting the visibilit y of package members hides the helper functions and typ es behind the package’s API, allow ing the package maintainer to change the imp lementation wit h confidence that no code outside the package will be affec ted. Restr icting visibilit y also hides variables so that clients can access and update them only through exp orted function s that preserve int ernal invar iants or enforce mut ual exclusion in a con cur rent program.

Wh en we change a file, we must recompi le the file’s package and pot ent ial ly all the packages that depend on it. Go compi lat ion is not ably faster than most other comp ile d languages, even when bui lding fro m scratch. There are three main reasons for the compi ler’s speed. First, all imports must be explicitly liste d at the beg inning of each source file, so the compi ler does not have to read and pro cess an ent ire file to deter mine its dependencies. Secon d, the dep endencies of a package for m a direc ted acyclic graph, and because there are no cyc les, packages can be compi led sep arately and perhaps in paral lel. Final ly, the obj e ct file for a compi led Go package records exp ort infor mat ion not just for the package its elf, but for its dep endencies too.

Wh en compi ling a package, the compi ler must read one obj e ct file for each imp ort but need not look beyon d thes e files.

10.2. Import Paths

ptg16091132

Each package is identified by a unique str ing cal le d its import pat h. Imp ort pat hs are the st rings that app ear in import de clarat ions.

import (

"fmt"

"math/rand"

"encoding/json"

"golang.org/x/net/html"

"github.com/go-sql-driver/mysql"

)

As we mention ed in Sec tion 2.6.1, the Go langu age specification doesn’t define the meaning of thes e st rings or how to deter mine a package’s imp ort pat h, but leaves these issues to the tools.

In this chapt er, we’ll take a det aile d lo ok at how the go to ol int erprets them, since that’s what the maj ority of Go programmers use for bui lding, testing , and so on. Other tools do exist, though. For example, Go programmers using Google’s int ernal multi-l angu age bui ld system fo llow dif ferent rules for naming and locat ing packages, specif ying tests, and so on, that more clos ely match the convent ion s of that system.

Fo r packages you int end to share or publish, imp ort pat hs should be globally unique. To avoid conflic ts, the imp ort pat hs of all packages other than those fro m the stand ard librar y should st art wit h the Int ernet domain name of the organizat ion that owns or hosts the package; this also makes it possible to find packages. For example, the declarat ion above imp orts an HTML

pars er maintained by the Go team and a popu lar third-p arty MySQL dat abas e dr iver.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 10.4. IMPORT DECLARATIONS

285

10.3. The Package Declaration

A package de clarat ion is required at the start of every Go source file. Its main pur pos e is to deter mine the defau lt identifier for that package (called the pack age nam e) when it is imp orted by another package.

Fo r example, every file of the math/rand package starts wit h package rand, so when you import this package, you can access its members as rand.Int, rand.Float64, and so on.

package main

import (

"fmt"

"math/rand"

)

func main() {

fmt.Println(rand.Int())

}

Conv ent ion ally, the package name is the last seg ment of the imp ort pat h, and as a result, two packages may have the same name even thoug h their import pat hs necessarily dif fer. For example, the packages whose imp ort pat hs are math/rand and crypto/rand both have the name rand. We’ll see how to use bot h in the same program in a mom ent.

ptg16091132

There are three maj or exception s to the ‘‘last seg ment’’ conv ent ion. The firs t is that a package defining a command (an exec utable Go program) always has the name main, regardless of the package’s imp ort pat h. This is a sig nal to go build (§10.7.3) that it must inv oke the lin ker to make an exe cut able file.

The secon d exception is that som e files in the direc tor y may have the suffix _test on their package name if the file name end s with _test.go. Such a direc tor y may define two packages: the usu al one, plu s anot her one cal le d an exter nal test package. The _test suffix sig nals to go test that it must bui ld both packages, and it indic ates which files belon g to each package.

External test packages are used to avoid cyc les in the imp ort graph arising fro m dep endencies of the test; the y are cov ere d in more det ail in Sec tion 11.2.4.

The third exception is that som e to ols for dep endency management app end version number suffixes to package imp ort pat hs, such as "gopkg.in/yaml.v2". The package name exc ludes the suffix, so in this cas e it wou ld be just yaml.

10.4. Import Declarations

A Go source file may cont ain zero or more import de clarat ions immediate ly after the package de clarat ion and before the firs t non-import declarat ion. Each imp ort declarat ion may specif y the imp ort pat h of a single package, or multiple packages in a parenthesize d list. The two forms below are equivalent but the secon d form is more common.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

286

CHAPTER 10. PACKAGES AND THE GO TOOL

import "fmt"

import "os"

import (

"fmt"

"os"

)

Imported packages may be gro uped by int roducing blank lines; such gro upings usu ally indi-cate dif ferent dom ains. The order is not sig nificant, but by convent ion the lines of each gro up are sor ted alp hab etic ally. (Both gofmt and goimports wi l l group and sor t for you .) import (

"fmt"

"html/template"

"os"

"golang.org/x/net/html"

"golang.org/x/net/ipv4"

)

If we need to imp ort two packages whose names are the same, like math/rand and crypto/rand, int o a third package, the imp ort declarat ion must specif y an alternat ive name for at least one of them to avoid a conflic t. This is cal le d a renami n g import.

import (

ptg16091132

"crypto/rand"

mrand "math/rand" // alternative name mrand avoids conflict

)

The alt ernat ive name affec ts only the imp orting file. Other files, even ones in the same package, may import the package using its default name, or a dif ferent name.

A renaming imp ort may be useful even when there is no conflic t. If the name of the imp orted package is unw ieldy, as is som etimes the cas e for aut omat ical ly generated code, an abbre viate d name may be more convenient. The same short name should be used con sistent ly to avoid conf usion. Cho osing an alt ernat ive name can help avoid conflic ts with common local variable names. For example, in a file wit h many local variables named path, we mig ht imp ort the st and ard "path" package as pathpkg.

Each imp ort declarat ion est ablishes a dep endency fro m the cur rent package to the imp orted package. The go build to ol rep orts an error if these dep endencies for m a cyc le.

10.5. Blank Imports

It is an error to imp ort a package int o a file but not refer to the name it defines wit hin that file.

Ho wever, on occ asion we must imp ort a package merely for the side effe cts of doing so: evaluat ion of the initializer expressions of its package-le vel var iables and exe cut ion of its init function s (§2.6.2). To sup press the ‘‘unus ed imp ort’’ er ror we wou ld other wis e encounter, we must us e a renaming imp ort in which the alt ernat ive name is _, the blank identifier. As usu al, the

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 10.5. BLANK IMPORTS

287

bl ank identifier can never be reference d.

import _ "image/png" // register PNG decoder

This is known as a bl ank imp ort. It is most often used to imp lement a compi le-t ime me chanism where by the main program can enable opt ion al features by blank-importing addition al packages. First we’ll see how to use it, then we’ll see how it wor ks.

The stand ard librar y’s image package exp orts a Decode func tion that reads bytes fro m an io.Reader, figures out which image for mat was used to encode the dat a, invo kes the appropriate decoder, then retur ns the resulting image.Image. Using image.Decode, it’s easy to bui ld a simple image converter that reads an image in one for mat and writes it out in anot her :

gopl.io/ch10/jpeg

// The jpeg command reads a PNG image from the standard input

// and writes it as a JPEG image to the standard output.

package main

import (

"fmt"

"image"

"image/jpeg"

_ "image/png" // register PNG decoder

ptg16091132

"io"

"os"

)

func main() {

if err := toJPEG(os.Stdin, os.Stdout); err != nil {

fmt.Fprintf(os.Stderr, "jpeg: %v\n", err)

os.Exit(1)

}

}

func toJPEG(in io.Reader, out io.Writer) error {

img, kind, err := image.Decode(in)

if err != nil {

return err

}

fmt.Fprintln(os.Stderr, "Input format =", kind) return jpeg.Encode(out, img, &jpeg.Options{Quality: 95})

}

If we feed the out put of gopl.io/ch3/mandelbrot (§3.3) to the converter program, it detec ts the PNG inp ut for mat and writes a JPEG version of Figure 3.3.

$ go build gopl.io/ch3/mandelbrot

$ go build gopl.io/ch10/jpeg

$ ./mandelbrot | ./jpeg >mandelbrot.jpg

Input format = png

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

288

CHAPTER 10. PACKAGES AND THE GO TOOL

No tice the blank imp ort of image/png. Wit hout that line, the program comp iles and lin ks as usual but can no lon g er recog nize or decode inp ut in PNG for mat: $ go build gopl.io/ch10/jpeg

$ ./mandelbrot | ./jpeg >mandelbrot.jpg

jpeg: image: unknown format

Here’s how it wor ks. The stand ard librar y prov ides deco ders for GIF, PNG, and JPEG, and us ers may provide others, but to keep exe cut ables smal l, decoders are not include d in an application unless explicitly requeste d. The image.Decode func tion con sults a table of sup por ted formats. Each ent ry in the table specifies four things: the name of the for mat; a str ing that is a prefix of all images encoded this way, used to detec t the encoding; a function Decode that de codes an enco ded image; and another function DecodeConfig that deco des on ly the image metadata, such as its size and color space. An ent ry is adde d to the table by cal ling image.RegisterFormat, typic ally fro m within the package initializer of the sup por ting package for each for mat, like this one in image/png:

package png // image/png

func Decode(r io.Reader) (image.Image, error)

func DecodeConfig(r io.Reader) (image.Config, error)

func init() {

const pngHeader = "\x89PNG\r\n\x1a\n"

image.RegisterFormat("png", pngHeader, Decode, DecodeConfig) ptg16091132

}

The effec t is that an app lic ation need only blank-import the package for the for mat it needs to make the image.Decode func tion able to decode it.

The database/sql package uses a simi lar mechanism to let users ins tal l ju st the dat abas e dr ivers the y ne e d. For example:

import (

"database/mysql"

_ "github.com/lib/pq"

// enable support for Postgres

_ "github.com/go-sql-driver/mysql" // enable support for MySQL

)

db, err = sql.Open("postgres", dbname) // OK

db, err = sql.Open("mysql", dbname)

// OK

db, err = sql.Open("sqlite3", dbname)

// returns error:

unknown driver "sqlite3"

Exercis e 10.1: Extend the jpeg prog ram so that it converts any sup por ted inp ut for mat to any output for mat, using image.Decode to detec t the inp ut for mat and a flag to selec t the out put format.

Exercis e 10.2: Define a gener ic archive file-reading function cap able of reading ZIP files (archive/zip) and POSIX tar files (archive/tar). Use a reg ist rat ion mechanism similar to the one des crib ed above so that sup por t for each file for mat can be plugged in using blank imports.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 10.6. PACKAGES AND NAMING

289

10.6. Packages and Naming

In this sec tion, we’ll offer som e advice on how to fol low Go’s distinc tive convent ion s for naming packages and their members.

Wh en cre ating a package, keep its name short, but not so short as to be crypt ic. The most frequently used packages in the stand ard librar y are named bufio, bytes, flag, fmt, http, io, json, os, sort, sync, and time.

Be des crip tive and unambiguous where possible. For example, don’t name a utilit y package util when a name such as imageutil or ioutil is specific yet still con cis e. Avoid cho osing package names that are commonly used for rel ate d lo cal variables, or you may comp el the package’s clients to use renaming imp orts, as wit h the path package.

Package names usu ally take the singu lar for m. The stand ard packages bytes, errors, and strings us e the plural to avoid hiding the cor responding pre declare d types and, in the cas e of go/types, to avoid conflic t with a key word.

Av oid package names that already have other connot ation s. For example, we originally used the name temp for the temperature conversion package in Sec tion 2.5, but that didn’t last lon g.

It was a ter r ible ide a becaus e ‘‘temp’’ is an almost univers al synonym for ‘‘temp orar y.’’ We went thro ugh a brief per iod wit h the name temperature, but that was too lon g and didn’t say what the package did. In the end, it became tempconv, which is shorter and paral lel wit h strconv.

ptg16091132

No w let’s tur n to the naming of package members. Since each reference to a member of anot her package uses a qualified identifier such as fmt.Println, the burden of des cribing the package member is bor ne equ ally by the package name and the memb er name. We need not ment ion the con cept of for matting in Println becaus e the package name fmt do es that already. When desig ning a package, con sider how the two par ts of a qualified identifier wor k together, not the memb er name alone. Here are som e ch arac ter ist ic examples: bytes.Equal

flag.Int

http.Get

json.Marshal

We can identify som e common naming patterns. The strings package provides a number of indep endent functions for manipu lat ing str ings:

package strings

func Index(needle, haystack string) int

type Replacer struct{ /* ... */ }

func NewReplacer(oldnew ...string) *Replacer

type Reader struct{ /* ... */ }

func NewReader(s string) *Reader

The word string do es not app ear in any of their names. Clients refer to them as strings.Index, strings.Replacer, and so on.

Ot her packages that we mig ht des crib e as single-ty pe packages, such as html/template and math/rand, exp ose one princip al dat a type plu s its met hods, and often a New func tion to create ins tances.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

290

CHAPTER 10. PACKAGES AND THE GO TOOL

package rand // "math/rand"

type Rand struct{ /* ... */ }

func New(source Source) *Rand

This can lead to rep etit ion, as in template.Template or rand.Rand, which is why the names of these kinds of packages are often especi ally short.

At the other extreme, there are packages like net/http that have a lot of names wit hout a lot of str ucture, because the y perfor m a complic ated task. Despit e having over twent y types and many more functions, the package’s most imp ortant members have the simplest names: Get, Post, Handle, Error, Client, Server.

10.7. The Go Tool

The rest of this chapt er con cer ns the go to ol, which is used for dow nlo ading, quer ying, formatting , building, testing , and ins tal ling packages of Go code.

The go to ol com bines the features of a diverse set of tools int o on e command set. It is a package manager (analogou s to apt or rpm) that answers quer ies ab out its inv ent ory of packages, comp utes their dependencies, and dow nlo ads them fro m remote version-cont rol systems. It is ptg16091132

a bui ld system that computes file dependencies and inv okes compi lers, assemb lers, and lin kers, although it is int ent ion ally less complete than the stand ard Unix make. And it is a test dr iver, as we will see in Chapt er 11.

Its command-line int erface uses the ‘‘Sw iss army knife’’ st yle, wit h ov er a dozen sub command s, som e of which we have already seen, like get, run, build, and fmt. You can run go help to see the index of its bui lt-in document ation, but for reference, we’ve liste d the most common ly used command s below:

$ go

...

build

compile packages and dependencies

clean

remove object files

doc

show documentation for package or symbol

env

print Go environment information

fmt

run gofmt on package sources

get

download and install packages and dependencies

install

compile and install packages and dependencies

list

list packages

run

compile and run Go program

test

test packages

version

print Go version

vet

run go tool vet on packages

Use "go help [command]" for more information about a command.

...

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 10.7. THE GO TOOL

291

To keep the need for configurat ion to a minimum, the go to ol relies heavi ly on convent ion s.

Fo r example, given the name of a Go source file, the tool can find its enclosing package, becaus e each direc tor y cont ains a single package and the imp ort pat h of a package corresp ond s to the direc tor y hierarchy in the wor ksp ace. Given the imp ort pat h of a package, the to ol can find the cor responding direc tor y in which it stores obj e ct files. It can also find the URL of the ser ver that hosts the source code rep ository.

10.7.1. Workspace Organization

The only configurat ion most users ever need is the GOPATH enviro nment var iable, which specifies the root of the wor ksp ace. When switching to a dif ferent wor ksp ace, users update the value of GOPATH. For ins tance, we set GOPATH to $HOME/gobook while wor king on this book: $ export GOPATH=$HOME/gobook

$ go get gopl.io/...

Af ter you dow nlo ad al l the programs for this book using the command above , your wor ksp ace wi l l cont ain a hierarchy like this one:

GOPATH/

src/

gopl.io/

.git/

ptg16091132

ch1/

helloworld/

main.go

dup/

main.go

...

golang.org/x/net/

.git/

html/

parse.go

node.go

...

bin/

helloworld

dup

pkg/

darwin_amd64/

...

GOPATH has three sub direc tor ies. The src su bdirec tor y holds source code. Each package resides in a direc tor y whos e name rel ative to $GOPATH/src is the package’s imp ort pat h, such as gopl.io/ch1/helloworld. Obs erve that a single GOPATH worksp ace cont ains multiple version-cont rol rep ositories beneath src, such as gopl.io or golang.org. The pkg su bdirec tor y is where the bui ld to ols store compi led packages, and the bin su bdirec tor y holds exe cut able prog rams like helloworld.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

292

CHAPTER 10. PACKAGES AND THE GO TOOL

A secon d enviro nment var iable, GOROOT, specifies the root direc tor y of the Go distr ibution, which provides all the packages of the stand ard librar y. The direc tor y st ruc ture beneath GOROOT resemb les that of GOPATH, so, for example, the source files of the fmt package reside in the $GOROOT/src/fmt direc tor y. Users never need to set GOROOT since, by defau lt, the go to ol wi l l us e the locat ion where it was inst alled.

The go env command prints the effec tive values of the env iro nment var iables relevant to the to olchain, including the defau lt values for the missing ones. GOOS sp ecifies the target operating system (for example, android, linux, darwin, or windows) and GOARCH sp ecifies the target processor arc hit ecture, such as amd64, 386, or arm. Alt hough GOPATH is the only var iable you mu st set, the others occ asionally app ear in our explanation s.

$ go env

GOPATH="/home/gopher/gobook"

GOROOT="/usr/local/go"

GOARCH="amd64"

GOOS="darwin"

...

10.7.2. Downloading Packages

Wh en using the go to ol, a package’s imp ort pat h indic ates not only where to find it in the local ptg16091132

worksp ace, but where to find it on the Int ernet so that go get can ret rie ve and update it.

The go get command can down load a single package or an ent ire subtree or rep ository using the ... notation, as in the pre vious sec tion. The tool als o comp utes and dow nlo ads all the dep endencies of the initial packages, which is why the golang.org/x/net/html package appeared in the wor ksp ace in the pre vious example.

Once go get has dow nlo ade d the packages, it bui lds them and then install s the librar ies and commands. We’ll look at the det ails in the next sec tion, but an example will show how st raig htfor ward the pro cess is. The firs t command below gets the golint to ol, which che cks for common sty le pro blems in Go source code. The secon d command runs golint on gopl.io/ch2/popcount from Sec tion 2.6.2. It helpf ully rep orts that we have for g ott en to wr ite a doc comment for the package:

$ go get github.com/golang/lint/golint

$ $GOPATH/bin/golint gopl.io/ch2/popcount

src/gopl.io/ch2/popcount/main.go:1:1:

package comment should be of the form "Package popcount ..."

The go get command has sup por t for popu lar code-hosting sit es like GitHu b, Bitbucket, and Launchpad and can make the appro priate requests to their version-cont rol systems. For less we ll-k now n sites, you may have to indic ate which version-cont rol pro tocol to use in the import pat h, such as Git or Mercurial. Run go help importpath for the det ails.

The direc tor ies that go get creates are true clients of the remot e repository, not just copies of the files, so you can use version-cont rol command s to see a dif f of local edits you’ve made or to

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 10.7. THE GO TOOL

293

up date to a dif ferent revision. For example, the golang.org/x/net direc tor y is a Git client: $ cd $GOPATH/src/golang.org/x/net

$ git remote -v

origin https://go.googlesource.com/net (fetch)

origin https://go.googlesource.com/net (push)

No tice that the app arent dom ain name in the package’s imp ort pat h, golang.org, dif fers fro m the actual dom ain name of the Git ser ver, go.googlesource.com. This is a feature of the go to ol that lets packages use a custom dom ain name in their import pat h while being hoste d by a generic ser vice such as googlesource.com or github.com.

HTML pages beneath

https://golang.org/x/net/html include the metadata shown below, which redirec ts the go to ol to the Git rep ository at the actual hosting sit e: $ go build gopl.io/ch1/fetch

$ ./fetch https://golang.org/x/net/html | grep go-import

<meta name="go-import"

content="golang.org/x/net git https://go.googlesource.com/net"> If you specif y the -u flag , go get wi l l ensure that all packages it visits, including dep endencies, are updated to their latest version before being bui lt and ins tal le d. Wit hout that flag , packages that already exist local ly will not be updated.

The go get -u command general ly ret rie ves the latest version of each package, which is convenient when you’re getting started but may be inappro priate for deploye d proj e cts, where ptg16091132

precis e cont rol of dep endencies is critical for release hyg iene. The usu al solut ion to this problem is to vend or the code, that is, to make a persistent local copy of all the necessary dep endencies, and to update this copy caref ully and delib erately. Prior to Go 1.5, this required ch ang ing those packages’ import pat hs, so our copy of golang.org/x/net/html would become gopl.io/vendor/golang.org/x/net/html. More recent versions of the go to ol supp ort vendoring direc tly, thoug h we don’t have space to show the det ails here. See Vend or Dire ctor ies in the out put of the go help gopath command.

Exercis e 10.3: Using fetch http://gopl.io/ch1/helloworld?go-get=1, find out which service hosts the code samples for this book. (HTTP requests fro m go get include the go-get parameter so that ser vers can dist inguish them fro m ordinar y browser requests.) 10.7.3. Building Packages

The go build command compi les each argument package. If the package is a librar y, the resu lt is dis carde d; this merely che cks that the package is fre e of compi le error s. If the package is name d main, go build invo kes the lin ker to cre ate an exe cut able in the cur rent direc tor y; the name of the exe cut able is taken fro m the last seg ment of the package’s imp ort pat h.

Since each direc tor y cont ains one package, each exe cut able program, or comm and in Unix ter-minolog y, requires its own direc tor y. These direc tor ies are som etimes chi ldren of a direc tor y name d cmd, such as the golang.org/x/tools/cmd/godoc command which ser ves Go package documentation through a web int erface (§10.7.4).

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

294

CHAPTER 10. PACKAGES AND THE GO TOOL

Packages may be specified by their import pat hs, as we saw abov e, or by a rel ative direc tor y name, which must start wit h a . or .. segment even if this wou ld not ordinar ily be required.

If no argument is provide d, the cur rent direc tor y is assume d. Thu s the fol low ing command s build the same package, thoug h each writes the exe cut able to the direc tor y in which go build is run:

$ cd $GOPATH/src/gopl.io/ch1/helloworld

$ go build

and:

$ cd anywhere

$ go build gopl.io/ch1/helloworld

and:

$ cd $GOPATH

$ go build ./src/gopl.io/ch1/helloworld

but not:

$ cd $GOPATH

$ go build src/gopl.io/ch1/helloworld

Error: cannot find package "src/gopl.io/ch1/helloworld".

Packages may also be specified as a list of file names, thoug h this tends to be used only for smal l prog rams and one-off exp eriments. If the package name is main, the exe cut able name ptg16091132

comes fro m the bas ename of the firs t .go file.

$ cat quoteargs.go

package main

import (

"fmt"

"os"

)

func main() {

fmt.Printf("%q\n", os.Args[1:])

}

$ go build quoteargs.go

$ ./quoteargs one "two three" four\ five

["one" "two three" "four five"]

Partic ularly for throwaway programs like this one, we want to run the exe cut able as soon as we’ve bui lt it. The go run command com bines these two steps: $ go run quoteargs.go one "two three" four\ five

["one" "two three" "four five"]

The firs t argument that doesn’t end in .go is assume d to be the beg inning of the list of arguments to the Go exe cut able.

By defau lt, the go build command bui lds the requeste d package and all its dependencies, then throws away all the compi led code except the final exe cut able, if any. Bot h the dep endency

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 10.7. THE GO TOOL

295

analysis and the compi lat ion are sur prisingly fast, but as pro jec ts grow to dozens of packages and hundreds of thous and s of lines of code, the time to recompi le dep endencies can becom e notice able, pot ent ial ly several secon ds, even when those dep endencies haven’t change d at all.

The go install command is ver y simi lar to go build, except that it saves the compi led code for each package and command ins tead of throw ing it away. Compi led packages are saved bene ath the $GOPATH/pkg direc tor y corresponding to the src direc tor y in which the source resides, and command exe cut ables are saved in the $GOPATH/bin direc tor y. (Many users put $GOPATH/bin on their exec utable searc h path.) Thereafter, go build and go install do not run the compi ler for those packages and command s if the y have not change d, mak ing subsequent bui lds much faster. For convenience, go build -i inst alls the packages that are dep endencies of the bui ld target.

Since compi led packages vary by platfor m and arc hit ecture, go install saves them beneath a su bdirec tor y whos e name incor porates the values of the GOOS and GOARCH enviro nment var iables. For example, on a Mac the golang.org/x/net/html package is compi led and ins tal le d in the file golang.org/x/net/html.a under $GOPATH/pkg/darwin_amd64.

It is straig htfor ward to cross-compi le a Go program, that is, to bui ld an exec utable int ended for a dif ferent operat ing system or CPU. Just set the GOOS or GOARCH var iables dur ing the bui ld.

The cross prog ram pr ints the operat ing system and arc hit ecture for which it was bui lt: gopl.io/ch10/cross

ptg16091132

func main() {

fmt.Println(runtime.GOOS, runtime.GOARCH)

}

The fol low ing command s produce 64-bit and 32-bit exe cut ables respectively: $ go build gopl.io/ch10/cross

$ ./cross

darwin amd64

$ GOARCH=386 go build gopl.io/ch10/cross

$ ./cross

darwin 386

Some packages may need to compi le dif ferent versions of the code for cer tain platfor ms or processors, to deal wit h low-le vel por tabilit y issues or to provide opt imize d versions of important routines, for ins tance. If a file name includes an operat ing system or pro cessor archit ecture name like net_linux.go or asm_amd64.s, then the go to ol will compi le the file on ly when bui lding for that target. Speci al comments cal le d buil d tags give more fine-g rained cont rol . Fo r example, if a file cont ains this comment:

// +build linux darwin

before the package declarat ion (and its doc comment), go build wi l l comp ile it only when building for Linux or Mac OS X, and this comment says never to compi le the file:

// +build ignore

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

296

CHAPTER 10. PACKAGES AND THE GO TOOL

Fo r more det ails, see the Bu ild Const rai nts section of the go/build package’s documentation: $ go doc go/build

10.7.4. Documenting Packages

Go sty le stron gly encourages good documentation of package APIs. Each declarat ion of an exp orted package member and the package declarat ion its elf should be immediate ly pre ceded by a comment explaining its pur pos e and usage .

Go doc comments are always comp lete sentences, and the firs t sent ence is usu ally a summar y that starts wit h the name being declare d. Function parameters and other identifiers are mention ed wit hout quotation or marku p. For example, here’s the doc comment for fmt.Fprintf:

// Fprintf formats according to a format specifier and writes to w.

// It returns the number of bytes written and any write error encountered.

func Fprintf(w io.Writer, format string, a ...interface{}) (int, error) The det ails of Fprintf’s for matting are explained in a doc comment ass oci ated wit h the fmt package its elf. A comment immediate ly pre ceding a package de clarat ion is con sidered the do c comment for the package as a whole. There must be only one, thoug h it may appear in any file. Lon g er package comments may warrant a file of their own; fmt’s is over 300 lines.

ptg16091132

This file is usu ally cal le d doc.go.

Go o d do cumentation need not be extensive , and documentation is no subst itute for simplicity.

In deed, Go’s convent ion s favor brevit y and simplicity in documentation as in all things, since do cumentation, like code, requires maintenance too. Many declarat ions can be explained in on e we ll-worde d sent ence, and if the beh avior is truly obv iou s, no comment is needed.

Throughout the book, as space per mits, we’ve pre ceded many declarat ions by doc comments, but you will find better examples as you brows e the stand ard librar y. Two tools can help you do that.

The go doc to ol prints the declarat ion and doc comment of the ent ity specified on the command line, which may be a package:

$ go doc time

package time // import "time"

Package time provides functionality for measuring and displaying time.

const Nanosecond Duration = 1 ...

func After(d Duration) <-chan Time

func Sleep(d Duration)

func Since(t Time) Duration

func Now() Time

type Duration int64

type Time struct { ... }

...many more...

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 10.7. THE GO TOOL

297

or a package member:

$ go doc time.Since

func Since(t Time) Duration

Since returns the time elapsed since t.

It is shorthand for time.Now().Sub(t).

or a met hod:

$ go doc time.Duration.Seconds

func (d Duration) Seconds() float64

Seconds returns the duration as a floating-point number of seconds.

The tool does not need complete imp ort pat hs or cor rec t identifier cas e. This command prints the documentation of (*json.Decoder).Decode from the encoding/json package: $ go doc json.decode

func (dec *Decoder) Decode(v interface{}) error

Decode reads the next JSON-encoded value from its input and stores it in the value pointed to by v.

The secon d to ol, confusingly named godoc, ser ves cross-lin ked HTML pages that provide the same infor mat ion as go doc and much more. The godoc server at https://golang.org/pkg covers the stand ard librar y. Figure 10.1 shows the documentation for the time package, and ptg16091132

in Sec tion 11.6 we’ll see godoc’s int erac tive displ ay of example programs. The godoc server at https://godoc.org has a searc hable index of thous and s of open-s ource packages.

Yo u can als o run an ins tance of godoc in your wor ksp ace if you want to brows e your own packages. Visit http://localhost:8000/pkg in your brows er while running this command: $ godoc -http :8000

Its -analysis=type and -analysis=pointer flags augment the documentation and the source code wit h the results of advance d st atic analysis.

10.7.5. Int ernal Packages

The package is the most imp ortant mechanism for enc apsulat ion in Go programs. Unexported identifiers are visible only wit hin the same package, and exp orted identifiers are visible to the wor ld.

Sometimes, thoug h, a midd le ground wou ld be helpf ul, a way to define identifiers that are visible to a small set of trusted packages, but not to everyon e. For example, when we’re bre aking up a large package int o more manageable par ts, we may not want to reveal the int erfaces between those par ts to other packages. Or we may want to share utilit y func tions across several packages of a pro jec t without exp osing them more widely. Or perhaps we just want to exp eriment wit h a new package wit hout premature ly committ ing to its API, by put ting it ‘‘on prob ation’’ with a limite d set of clients.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

298

CHAPTER 10. PACKAGES AND THE GO TOOL

ptg16091132

Figure 10.1. The time package in godoc.

To address these needs, the go build to ol tre ats a package speci ally if its import pat h cont ains a pat h segment named internal. Such packages are cal le d internal packages. An int ernal package may be imp orted only by another package that is inside the tre e ro ote d at the parent of the internal direc tor y. For example, given the packages below, net/http/internal/chunked can be imp orted fro m net/http/httputil or net/http, but not fro m net/url. How ever, net/url may imp ort net/http/httputil.

net/http

net/http/internal/chunked

net/http/httputil

net/url

10.7.6. Querying Packages

The go list to ol rep orts infor mat ion about avai lable packages. In its simplest for m, go list tests whether a package is present in the wor ksp ace and prints its import pat h if so: $ go list github.com/go-sql-driver/mysql

github.com/go-sql-driver/mysql

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 10.7. THE GO TOOL

299

An argument to go list may contain the ‘‘...’’ wi ldc ard, which match es any subst ring of a package’s imp ort pat h. We can use it to enumerate all the packages wit hin a Go wor ksp ace: $ go list ...

archive/tar

archive/zip

bufio

bytes

cmd/addr2line

cmd/api

...many more...

or wit hin a specific subtree:

$ go list gopl.io/ch3/...

gopl.io/ch3/basename1

gopl.io/ch3/basename2

gopl.io/ch3/comma

gopl.io/ch3/mandelbrot

gopl.io/ch3/netflag

gopl.io/ch3/printints

gopl.io/ch3/surface

or rel ate d to a par tic ular topic:

ptg16091132

$ go list ...xml...

encoding/xml

gopl.io/ch7/xmlselect

The go list command obt ains the complete met adat a for each package, not just the imp ort path, and makes this infor mat ion avai lable to users or other tools in a var iety of for mats. The

-json flag causes go list to print the ent ire record of each package in JSON for mat: $ go list -json hash

{

"Dir": "/home/gopher/go/src/hash",

"ImportPath": "hash",

"Name": "hash",

"Doc": "Package hash provides interfaces for hash functions.",

"Target": "/home/gopher/go/pkg/darwin_amd64/hash.a",

"Goroot": true,

"Standard": true,

"Root": "/home/gopher/go",

"GoFiles": [

"hash.go"

],

"Imports": [

"io"

],

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

300

CHAPTER 10. PACKAGES AND THE GO TOOL

"Deps": [

"errors",

"io",

"runtime",

"sync",

"sync/atomic",

"unsafe"

]

}

The -f flag lets users customize the out put for mat using the templ ate langu age of package text/template (§4.6). This command prints the transitive dep endencies of the strconv package, sep arated by spaces:

$ go list -f '{{join .Deps " "}}' strconv

errors math runtime unicode/utf8 unsafe

and this command prints the direc t imports of each package in the compress su btree of the st and ard librar y:

$ go list -f '{{.ImportPath}} -> {{join .Imports " "}}' compress/...

compress/bzip2 -> bufio io sort

compress/flate -> bufio fmt io math sort strconv

compress/gzip -> bufio compress/flate errors fmt hash hash/crc32 io time compress/lzw -> bufio errors fmt io

ptg16091132

compress/zlib -> bufio compress/flate errors fmt hash hash/adler32 io The go list command is useful for bot h on e-off int erac tive quer ies and for bui ld and test automation scr ipts. We’ll use it again in Sec tion 11.2.4. For more infor mat ion, including the set of avai lable fields and their meaning, see the out put of go help list.

In this chapt er, we’ve explained all the imp ortant sub command s of the go to ol—except one. In the next chapt er, we’ll see how the go test command is used for testing Go programs.

Exercis e 10.4: Cons truct a tool that rep orts the set of all packages in the wor ksp ace that transitive ly dep end on the packages specified by the arguments. Hint: you will need to run go list twice, once for the initial packages and once for all packages. You may want to parse its JSON out put using the encoding/json package (§4.5).

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

11

Testing

Maur ice Wi l kes, the develop er of EDSAC, the firs t stored-program comp uter, had a startling insig ht whi le climbing the stairs of his lab orator y in 1949. In Memoir s of a Computer Pione er, he recal le d, ‘‘The realizat ion came over me wit h full force that a good par t of the remainder of my life was going to be spent in finding error s in my own programs.’’ Sure ly every program-ptg16091132

mer of a stored-program comp uter since then can symp athize wit h Wi l kes, thoug h perh aps not wit hout som e bemu sement at his naïveté about the difficult ies of sof tware con str uction.

Prog rams today are far larger and more complex than in Wilkes’s time, of cours e, and a gre at de al of effor t has been spent on techniques to make this complexity manageable. Two te chniques in par tic ular stand out for their effe ctiveness. The firs t is routine peer revie w of prog rams before the y are deploye d. The secon d, the subjec t of this chapt er, is testing .

Test ing , by which we imp licitly mean au tom ate d test ing , is the prac tice of writing small prog rams that che ck that the code under test (the produc tion co de) behaves as exp ected for cer tain inputs, which are usu ally eit her caref ully chosen to exercis e cer tain features or randomize d to ens ure bro ad cov erage.

The field of sof tware testing is enormou s. The task of testing occ upies all programmers som e of the time and som e prog rammers all of the time. The lit erature on testing includes thou sands of print ed books and millions of words of blog posts. In every mainst ream prog ramming langu age, there are dozens of sof tware packages intended for test con str uction, some wit h a gre at deal of the ory, and the field seems to att rac t more than a few pro phets wit h cult-li ke fol low ings. It is almost enoug h to convince programmers that to write effec tive tests they must acquire a whole new set of ski l ls.

Go’s appro ach to test ing can seem rat her low-tech in comparison. It relies on one command, go test, and a set of convent ion s for writing test functions that go test can run. The comparat ive ly lig htweig ht mechanism is effec tive for pure testing , and it extends natural ly to benchmarks and systematic examples for documentation.

301

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

302

CHAPTER 11. TESTING

In prac tice, writing test code is not much dif ferent fro m wr iting the original program its elf.

We write short functions that focus on one par t of the task. We have to be caref ul of bound ary condit ion s, thin k ab out dat a st ruc tures, and reason about what results a comp utation should produce fro m suit able inp uts. But this is the same pro cess as writing ordinar y Go code; it ne e dn’t require new not ation s, convent ion s, and tools.

11.1. The go test Tool

The go test su bcommand is a test driver for Go packages that are organize d according to certain conv ent ion s. In a package direc tor y, files whose names end wit h _test.go are not par t of the package ordinar ily bui lt by go build but are a par t of it when bui lt by go test.

Wi thin *_test.go files, three kinds of functions are tre ate d sp eci ally : tests, benchmarks, and examples. A te st func tion, which is a function whose name beg ins wit h Test, exercis es som e prog ram log ic for cor rec t behavior ; go test calls the test function and rep orts the result, which is eit her PASS or FAIL. A benchm ark fun cti on has a name beg inning wit h Benchmark and measures the per for mance of som e op erat ion; go test reports the mean exec ution time of the operat ion. And an ex amp le fun cti on, whose name starts wit h Example, provides machine-che cke d do cumentation. We will cov er tests in detai l in Sec tion 11.2, benchmarks in Section 11.4, and examples in Sec tion 11.6.

ptg16091132

The go test to ol scans the *_test.go files for these speci al func tions, generates a temporar y main package that cal ls them all in the pro per way, bui lds and runs it, rep orts the results, and then cle ans up.

11.2. Test Functions

Each test file must imp ort the testing package. Test functions have the fol low ing sig nature: func TestName(t *testing.T) {

// ...

}

Test function names must beg in with Test; the opt ion al suffix Name mu st beg in with a capit al letter:

func TestSin(t *testing.T) { /* ... */ }

func TestCos(t *testing.T) { /* ... */ }

func TestLog(t *testing.T) { /* ... */ }

The t parameter provides met hods for rep orting test fai lures and log ging addition al infor mat ion. Let’s define an example package gopl.io/ch11/word1, containing a single function IsPalindrome that rep orts whether a str ing reads the same for ward and backward. (This implementation tests every byte twice if the str ing is a palindrom e; we’ll com e back to that shortly.)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 11.2. TEST FUNCTIONS

303

gopl.io/ch11/word1

// Package word provides utilities for word games.

package word

// IsPalindrome reports whether s reads the same forward and backward.

// (Our first attempt.)

func IsPalindrome(s string) bool {

for i := range s {

if s[i] != s[len(s)-1-i] {

return false

}

}

return true

}

In the same direc tor y, the file word_test.go cont ains two test functions named TestPalindrome and TestNonPalindrome. Each che cks that IsPalindrome gives the rig ht answer for a single inp ut and rep orts fai lures using t.Error:

package word

import "testing"

func TestPalindrome(t *testing.T) {

ptg16091132

if !IsPalindrome("detartrated") {

t.Error(ÌsPalindrome("detartrated") = falsè)

}

if !IsPalindrome("kayak") {

t.Error(ÌsPalindrome("kayak") = falsè)

}

}

func TestNonPalindrome(t *testing.T) {

if IsPalindrome("palindrome") {

t.Error(ÌsPalindrome("palindrome") = truè)

}

}

A go test (or go build) command wit h no package arguments operates on the package in the cur rent direc tor y. We can bui ld and run the tests wit h the fol low ing command.

$ cd $GOPATH/src/gopl.io/ch11/word1

$ go test

ok

gopl.io/ch11/word1

0.008s

Sat isfied, we ship the program, but no sooner have the launch par ty guests depar ted than the bug rep orts start to arrive . A French user named Noelle Eve Elleon compl ains that IsPalindrome do esn’t recog nize ‘‘été.’’ Anot her, fro m Cent ral America, is dis app ointe d that it rej e cts

‘‘A man, a plan, a canal: Panama.’’ Thes e sp ecific and small bug rep orts natural ly lend themselves to new test cas es.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

304

CHAPTER 11. TESTING

func TestFrenchPalindrome(t *testing.T) {

if !IsPalindrome("été") {

t.Error(ÌsPalindrome("été") = falsè)

}

}

func TestCanalPalindrome(t *testing.T) {

input := "A man, a plan, a canal: Panama"

if !IsPalindrome(input) {

t.Errorf(ÌsPalindrome(%q) = falsè, input)

}

}

To avoid writing the lon g input st ring twice, we use Errorf, which provides for matting like Printf.

Wh en the two new tests have been adde d, the go test command fai ls wit h infor mat ive error mess ages.

$ go test

--- FAIL: TestFrenchPalindrome (0.00s)

word_test.go:28: IsPalindrome("été") = false

--- FAIL: TestCanalPalindrome (0.00s)

word_test.go:35: IsPalindrome("A man, a plan, a canal: Panama") = false FAIL

ptg16091132

FAIL

gopl.io/ch11/word1

0.014s

It’s good prac tice to write the test firs t and obs erve that it triggers the same fai lure des crib ed by the user’s bug rep ort. Only then can we be confident that whate ver fix we com e up wit h addresses the rig ht pro blem.

As a bonus, running go test is usually quicker than manually going through the steps des crib ed in the bug rep ort, allow ing us to iterate more rapid ly. If the test suite contains many slow tests, we may make even faster progress if we’re selec tive about which ones we run.

The -v flag prints the name and exe cut ion time of each test in the package: $ go test -v

=== RUN TestPalindrome

--- PASS: TestPalindrome (0.00s)

=== RUN TestNonPalindrome

--- PASS: TestNonPalindrome (0.00s)

=== RUN TestFrenchPalindrome

--- FAIL: TestFrenchPalindrome (0.00s)

word_test.go:28: IsPalindrome("été") = false

=== RUN TestCanalPalindrome

--- FAIL: TestCanalPalindrome (0.00s)

word_test.go:35: IsPalindrome("A man, a plan, a canal: Panama") = false FAIL

exit status 1

FAIL

gopl.io/ch11/word1

0.017s

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 11.2. TEST FUNCTIONS

305

and the -run flag , whos e argument is a regu lar expression, causes go test to run only those tests whose function name match es the pattern:

$ go test -v -run="French|Canal"

=== RUN TestFrenchPalindrome

--- FAIL: TestFrenchPalindrome (0.00s)

word_test.go:28: IsPalindrome("été") = false

=== RUN TestCanalPalindrome

--- FAIL: TestCanalPalindrome (0.00s)

word_test.go:35: IsPalindrome("A man, a plan, a canal: Panama") = false FAIL

exit status 1

FAIL

gopl.io/ch11/word1

0.014s

Of course, once we’ve gott en the selec ted tests to pass, we should inv oke go test with no flags to run the ent ire test suite one last time before we commit the change .

No w our task is to fix the bugs. A quick inv est igat ion reveals the cause of the firs t bug to be IsPalindrome’s use of byte sequences, not rune sequences, so that non-ASCII charac ters such as the é in "été" conf use it. The secon d bug arises fro m not ignor ing spaces, punctuation, and letter cas e.

Chastene d, we rewrite the function more caref ully:

ptg16091132

gopl.io/ch11/word2

// Package word provides utilities for word games.

package word

import "unicode"

// IsPalindrome reports whether s reads the same forward and backward.

// Letter case is ignored, as are non-letters.

func IsPalindrome(s string) bool {

var letters []rune

for _, r := range s {

if unicode.IsLetter(r) {

letters = append(letters, unicode.ToLower(r))

}

}

for i := range letters {

if letters[i] != letters[len(letters)-1-i] {

return false

}

}

return true

}

We als o wr ite a more comprehensive set of test cas es that combines all the pre vious ones and a numb er of new ones int o a table.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

306

CHAPTER 11. TESTING

func TestIsPalindrome(t *testing.T) {

var tests = []struct {

input string

want bool

}{

{"", true},

{"a", true},

{"aa", true},

{"ab", false},

{"kayak", true},

{"detartrated", true},

{"A man, a plan, a canal: Panama", true},

{"Evil I did dwell; lewd did I live.", true},

{"Able was I ere I saw Elba", true},

{"été", true},

{"Et se resservir, ivresse reste.", true},

{"palindrome", false}, // non-palindrome

{"desserts", false},

// semi-palindrome

}

for _, test := range tests {

if got := IsPalindrome(test.input); got != test.want {

t.Errorf("IsPalindrome(%q) = %v", test.input, got)

}

}

ptg16091132

}

Our new tests pass:

$ go test gopl.io/ch11/word2

ok

gopl.io/ch11/word2

0.015s

This sty le of tabl e-driven test ing is ver y common in Go. It is straig htfor ward to add new table entr ies as needed, and since the ass ertion log ic is not dup lic ated, we can invest more effor t in producing a good error message .

The out put of a fai ling test does not include the ent ire stack trace at the moment of the cal l to t.Errorf. Nor does t.Errorf caus e a panic or stop the exe cut ion of the test, unli ke ass ertion fai lures in many test framewor ks for other langu ages. Tests are indep endent of each other. If an early ent ry in the table causes the test to fai l, later table entr ies will still be che cke d, and thus we may lear n ab out multiple fai lures dur ing a single run.

Wh en we really must stop a test function, perhaps because some initializat ion code fai le d or to prevent a fai lure already rep orted fro m causing a confusing cas cade of others, we use t.Fatal or t.Fatalf. These must be cal le d from the same goroutine as the Test func tion, not fro m anot her one cre ate d during the test.

Test fai lure messages are usu ally of the for m "f(x) = y, want z", where f(x) explains the attemp ted operat ion and its input, y is the actual result, and z the exp ected result. Where convenient, as in our palindrom e example, actual Go syntax is used for the f(x) part. Displ aying x is par tic ularly imp ortant in a table-dr iven test, since a given assertion is exe cut ed many

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 11.2. TEST FUNCTIONS

307

times wit h dif ferent values. Avoid boi ler plate and redundant infor mat ion. When testing a boole an func tion such as IsPalindrome, omit the want z part since it adds no infor mat ion. If x, y, or z is lengt hy, print a con cis e summar y of the relevant par ts instead. The aut hor of a test should str ive to help the programmer who must diagnos e a test fai lure.

Exercis e 11.1: Wr ite tests for the charcount prog ram in Sec tion 4.3.

Exercis e 11.2: Wr ite a set of tests for IntSet (§6.5) that che cks that its beh avior after each op erat ion is equivalent to a set bas ed on bui lt-in maps. Save your imp lementation for benchmarking in Exercis e 11.7.

11.2.1. Randomized Testing

Table-dr iven tests are convenient for che cking that a function wor ks on inp uts caref ully sele cte d to exercis e interest ing cas es in the log ic. Anot her approach, ra ndomized testing, explores a bro ader range of inp uts by con str ucting inp uts at random.

Ho w do we know what out put to exp ect fro m our function, given a random inp ut? There are two strateg ies. The firs t is to write an alt ernat ive imp lementation of the function that uses a less efficient but simpler and cle arer algor it hm, and che ck that bot h implementation s give the ptg16091132

same result. The secon d is to cre ate inp ut values according to a pattern so that we know what output to exp ect.

The example below uses the secon d approach: the randomPalindrome func tion generates word s that are known to be palindrom es by con str uction.

import "math/rand"

// randomPalindrome returns a palindrome whose length and contents

// are derived from the pseudo-random number generator rng.

func randomPalindrome(rng *rand.Rand) string {

n := rng.Intn(25) // random length up to 24

runes := make([]rune, n)

for i := 0; i < (n+1)/2; i++ {

r := rune(rng.Intn(0x1000)) // random rune up to '\u0999'

runes[i] = r

runes[n-1-i] = r

}

return string(runes)

}

func TestRandomPalindromes(t *testing.T) {

// Initialize a pseudo-random number generator.

seed := time.Now().UTC().UnixNano()

t.Logf("Random seed: %d", seed)

rng := rand.New(rand.NewSource(seed))

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

308

CHAPTER 11. TESTING

for i := 0; i < 1000; i++ {

p := randomPalindrome(rng)

if !IsPalindrome(p) {

t.Errorf("IsPalindrome(%q) = false", p)

}

}

}

Since randomize d tests are non deter minist ic, it is critical that the log of the fai ling test record sufficient infor mat ion to repro duce the fai lure. In our example, the inp ut p to IsPalindrome te lls us all we need to know, but for functions that accept more complex inputs, it may be simpler to log the seed of the pseudo-random number generator (as we do above) than to dump the ent ire inp ut dat a st ruc ture. Armed wit h that seed value, we can easi ly modif y the test to repl ay the fai lure deter minist ically.

By using the cur rent time as a source of randomness, the test will explore nov el inp uts each time it is run, over the ent ire cours e of its lifet ime. This is esp eci ally valuable if your pro jec t us es an aut omated system to run all its tests per iodic ally.

Exercis e 11.3: TestRandomPalindromes on ly tests palindrom es. Write a randomize d test that generates and ver ifies non-palindrom es.

Exercis e 11.4: Mo dif y randomPalindrome to exercis e IsPalindrome’s handling of punc-ptg16091132

tu ation and spaces.

11.2.2. Testing a Command

The go test to ol is useful for testing librar y packages, but wit h a lit tle effort we can use it to test command s as wel l. A package named main ordinar ily pro duces an exec utable program, but it can be imp orted as a librar y to o.

Let’s write a test for the echo prog ram of Sec tion 2.3.2. We’ve split the program into two function s: echo do es the real wor k, whi le main pars es and reads the flag values and rep orts any er ror s returned by echo.

gopl.io/ch11/echo

// Echo prints its command-line arguments.

package main

import (

"flag"

"fmt"

"io"

"os"

"strings"

)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 11.2. TEST FUNCTIONS

309

var (

n = flag.Bool("n", false, "omit trailing newline") s = flag.String("s", " ", "separator")

)

var out io.Writer = os.Stdout // modified during testing func main() {

flag.Parse()

if err := echo(!*n, *s, flag.Args()); err != nil {

fmt.Fprintf(os.Stderr, "echo: %v\n", err)

os.Exit(1)

}

}

func echo(newline bool, sep string, args []string) error {

fmt.Fprint(out, strings.Join(args, sep))

if newline {

fmt.Fprintln(out)

}

return nil

}

Fr om the test, we will cal l echo with a var iety of arguments and flag settings and che ck that it pr ints the cor rec t output in each cas e, so we’ve adde d parameters to echo to reduce its depen-ptg16091132

dence on global var iables. That said, we’ve als o introduce d anot her global var iable, out, the io.Writer to which the result will be writt en. By hav ing echo wr ite through this var iable, not direc tly to os.Stdout, the tests can subst itute a dif ferent Writer implementation that records what was writt en for later insp ection. Here’s the test, in file echo_test.go: package main

import (

"bytes"

"fmt"

"testing"

)

func TestEcho(t *testing.T) {

var tests = []struct {

newline bool

sep

string

args

[]string

want

string

}{

{true, "", []string{}, "\n"},

{false, "", []string{}, ""},

{true, "\t", []string{"one", "two", "three"}, "one\ttwo\tthree\n"},

{true, ",", []string{"a", "b", "c"}, "a,b,c\n"},

{false, ":", []string{"1", "2", "3"}, "1:2:3"},

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

310

CHAPTER 11. TESTING

for _, test := range tests {

descr := fmt.Sprintf("echo(%v, %q, %q)",

test.newline, test.sep, test.args)

out = new(bytes.Buffer) // captured output

if err := echo(test.newline, test.sep, test.args); err != nil {

t.Errorf("%s failed: %v", descr, err)

continue

}

got := out.(*bytes.Buffer).String()

if got != test.want {

t.Errorf("%s = %q, want %q", descr, got, test.want)

}

}

}

No tice that the test code is in the same package as the pro duc tion code. Alt hough the package name is main and it defines a main func tion, dur ing testing this package acts as a librar y that exp oses the function TestEcho to the test driver; its main func tion is ignored.

By organizing the test as a table, we can easi ly add new test cas es. Let’s see what happens when the test fai ls, by adding this line to the table:

{true, ",", []string{"a", "b", "c"}, "a b c\n"}, // NOTE: wrong expectation!

go test pr ints

ptg16091132

$ go test gopl.io/ch11/echo

--- FAIL: TestEcho (0.00s)

echo_test.go:31: echo(true, ",", ["a" "b" "c"]) = "a,b,c", want "a b c\n"

FAIL

FAIL

gopl.io/ch11/echo

0.006s

The error message des crib es the att emp ted operat ion (using Go-li ke syntax), the actual beh avior, and the exp ected beh avior, in that order. Wit h an infor mat ive error message such as this, you may have a prett y go o d ide a ab out the root cause before you’ve even located the source co de of the test.

It’s imp ortant that code being teste d not cal l log.Fatal or os.Exit, since these will stop the process in its tracks; cal ling these functions should be regarde d as the exc lusive rig ht of main.

If som ething tot ally unexp ected happens and a function panics, the test driver will recov er, though the test will of cours e be con sidered a fai lure. Exp ected error s such as those resulting from bad user inp ut, missing files, or imp rop er configurat ion should be rep orted by retur ning a non-ni l error value. For tunately (though unfor tunate as an illustrat ion), our echo example is so simple that it will never retur n a non-ni l er ror.

11.2.3. White-Box Testing

One way of categor izing tests is by the level of knowledge the y re quire of the int ernal wor kings of the package under test. A bl ack-box test assumes not hing about the package other than

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 11.2. TEST FUNCTIONS

311

what is exp osed by its API and specified by its document ation; the package’s int ernals are op aque. In contrast, a whit e-box test has privi leged access to the int ernal functions and dat a st ruc tures of the package and can make obs ervat ions and changes that an ordinar y client cannot. For example, a white-b ox test can che ck that the invar iants of the package’s dat a types are maintained after every operat ion. (The name whit e box is tradition al, but cl ear box would be more acc urate.)

The two appro ach es are complementary. Black-b ox tests are usu ally more robust, needing fe wer updates as the sof tware evo l ves. They als o help the test aut hor emp athize wit h the client of the package and can reveal flaws in the API desig n. In contrast, white-b ox tests can provide more det aile d coverage of the trick ier par ts of the imp lementation.

We’ve already seen examples of bot h kind s. TestIsPalindrome calls only the exp orted function IsPalindrome and is thu s a black-b ox test. TestEcho calls the echo func tion and up dates the global var iable out, bot h of which are unexp orted, mak ing it a white-b ox test.

While developing TestEcho, we modified the echo func tion to use the package-le vel var iable out when writing its out put, so that the test cou ld replace the stand ard out put wit h an alternative imp lementation that records the dat a for later insp ection. Using the same technique, we can replace other par ts of the pro duc tion code wit h easy-to-test ‘‘fake’’ implementation s.

The advantage of fake imp lementation s is that the y can be simpler to configure, more predic table, more reliable, and easier to obs erve . They can also avoid undesirable side effe cts ptg16091132

such as updat ing a pro duc tion dat abas e or charg ing a cre dit card.

The code below shows the quota-che cking log ic in a web ser vice that provides net wor ked storage to users. When users exceed 90% of their quota, the system sends them a war ning emai l.

gopl.io/ch11/storage1

package storage

import (

"fmt"

"log"

"net/smtp"

)

func bytesInUse(username string) int64 { return 0 /* ... */ }

// Email sender configuration.

// NOTE: never put passwords in source code!

const sender = "notifications@example.com"

const password = "correcthorsebatterystaple"

const hostname = "smtp.example.com"

const template = `Warning: you are using %d bytes of storage,

%d%% of your quota.`

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

312

CHAPTER 11. TESTING

func CheckQuota(username string) {

used := bytesInUse(username)

const quota = 1000000000 // 1GB

percent := 100 * used / quota

if percent < 90 {

return // OK

}

msg := fmt.Sprintf(template, used, percent)

auth := smtp.PlainAuth("", sender, password, hostname) err := smtp.SendMail(hostname+":587", auth, sender,

[]string{username}, []byte(msg))

if err != nil {

log.Printf("smtp.SendMail(%s) failed: %s", username, err)

}

}

We’d like to test it, but we don’t want the test to send out real email. So we mov e the email log ic into its own function and store that function in an unexp orted package-le vel var iable, notifyUser.

gopl.io/ch11/storage2

var notifyUser = func(username, msg string) {

auth := smtp.PlainAuth("", sender, password, hostname) err := smtp.SendMail(hostname+":587", auth, sender, ptg16091132

[]string{username}, []byte(msg))

if err != nil {

log.Printf("smtp.SendEmail(%s) failed: %s", username, err)

}

}

func CheckQuota(username string) {

used := bytesInUse(username)

const quota = 1000000000 // 1GB

percent := 100 * used / quota

if percent < 90 {

return // OK

}

msg := fmt.Sprintf(template, used, percent)

notifyUser(username, msg)

}

We can now write a test that subst itutes a simple fake not ification mechanism instead of sending real email. This one records the notified user and the contents of the mess age .

package storage

import (

"strings"

"testing"

)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 11.2. TEST FUNCTIONS

313

func TestCheckQuotaNotifiesUser(t *testing.T) {

var notifiedUser, notifiedMsg string

notifyUser = func(user, msg string) {

notifiedUser, notifiedMsg = user, msg

}

// ...simulate a 980MB-used condition...

const user = "joe@example.org"

CheckQuota(user)

if notifiedUser == "" && notifiedMsg == "" {

t.Fatalf("notifyUser not called")

}

if notifiedUser != user {

t.Errorf("wrong user (%s) notified, want %s", notifiedUser, user)

}

const wantSubstring = "98% of your quota"

if !strings.Contains(notifiedMsg, wantSubstring) {

t.Errorf("unexpected notification message <<%s>>, "+

"want substring %q", notifiedMsg, wantSubstring)

}

}

There’s one pro blem: after this test function has retur ned, CheckQuota no lon g er wor ks as it ptg16091132

should because it’s still using the test’s fake imp lementation of notifyUsers. (There is always a risk of this kind when updat ing global var iables.) We must modif y the test to restore the previous value so that subsequent tests obs erve no effec t, and we must do this on all exe cut ion paths, including test fai lures and panics. This natural ly sug gests defer.

func TestCheckQuotaNotifiesUser(t *testing.T) {

// Save and restore original notifyUser.

saved := notifyUser

defer func() { notifyUser = saved }()

// Install the test's fake notifyUser.

var notifiedUser, notifiedMsg string

notifyUser = func(user, msg string) {

notifiedUser, notifiedMsg = user, msg

}

// ...rest of test...

}

This pattern can be used to temporar ily save and restore all kinds of global var iables, including command-line flags, debugging opt ion s, and per for mance parameters; to ins tal l and remov e ho oks that cause the pro duc tion code to cal l some test code when som ething int erest ing happens; and to coax the pro duc tion code int o rare but imp ortant states, such as timeouts, erro rs, and even specific int erleavings of con cur rent activ ities.

Using global var iables in this way is safe only because go test do es not nor mal ly run multiple tests con cur rently.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

314

CHAPTER 11. TESTING

11.2.4. External Test Packages

Consider the packages net/url, which provides a URL parser, and net/http, which provides a web ser ver and HTTP client librar y. As we mig ht exp ect, the hig her-le vel net/http dep ends on the low er-le vel net/url. How ever, one of the tests in net/url is an example demon strat-ing the int erac tion bet ween URLs and the HTTP client librar y. In other words, a test of the lower-le vel package imp orts the hig her-le vel package.

Figure 11.1. A test of net/url dep ends on net/http.

Decl aring this test function in the net/url package wou ld cre ate a cyc le in the package import graph, as depic ted by the upwards arrow in Figure 11.1, but as we explained in Section 10.1, the Go specification forbids imp ort cyc les.

We res olve the pro blem by declaring the test function in an exter nal test package, that is, in a ptg16091132

file in the net/url direc tor y whos e package declarat ion reads package url_test. The ext ra suffix _test is a sig nal to go test that it should bui ld an addition al package containing just thes e files and run its tests. It may be helpf ul to thin k of this exter nal test package as if it had the imp ort pat h net/url_test, but it cannot be imp orted under this or any other name.

Becaus e exter nal tests live in a sep arate package, the y may imp ort helper packages that als o dep end on the package being teste d; an in-p ackage test cannot do this. In ter ms of the desig n layers, the exter nal test package is log ical ly hig her up than bot h of the packages it dep ends up on, as shown in Figure 11.2.

Figure 11.2. External test packages bre ak dep endency cycles.

By avoiding imp ort cyc les, exter nal test packages allow tests, especi ally inte grati on tests (w hich test the int erac tion of several comp onents), to imp ort other packages fre ely, exac tly as an applic ation wou ld.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 11.2. TEST FUNCTIONS

315

We can use the go list to ol to summar ize which Go source files in a package direc tor y are produc tion code, in-p ackage tests, and exter nal tests. We’ll use the fmt package as an example.

GoFiles is the list of files that contain the pro duc tion code; these are the files that go build wi l l include in your app lic ation:

$ go list -f={{.GoFiles}} fmt

[doc.go format.go print.go scan.go]

TestGoFiles is the list of files that als o belong to the fmt package, but these files, whose names all end in _test.go, are include d on ly when bui lding tests: $ go list -f={{.TestGoFiles}} fmt

[export_test.go]

The package’s tests wou ld usu ally reside in these files, thoug h unusually fmt has non e; we’ll explain the pur pos e of export_test.go in a mom ent.

XTestGoFiles is the list of files that con stitute the exter nal test package, fmt_test, so these files must imp ort the fmt package in order to use it. Again, the y are include d on ly dur ing testing:

$ go list -f={{.XTestGoFiles}} fmt

[fmt_test.go scan_test.go stringer_test.go]

ptg16091132

Sometimes an exter nal test package may need privi leged access to the int ernals of the package under test, if for example a white-b ox test must live in a sep arate package to avoid an import cycle. In such cas es, we use a trick: we add decl arat ions to an in-p ackage _test.go file to exp ose the necessary int ernals to the exter nal test. This file thu s of fers the test a ‘‘back door’’

to the package. If the source file exists only for this pur pos e and contains no tests its elf, it is of ten cal le d export_test.go.

Fo r example, the imp lementation of the fmt package needs the functionality of unicode.IsSpace as par t of fmt.Scanf. To avoid creating an undesirable dependency, fmt do es not import the unicode package and its large tables of dat a; instead, it contains a simpler implement ation, which it cal ls isSpace.

To ens ure that the beh avior s of fmt.isSpace and unicode.IsSpace do not drift apart, fmt pr udently contains a test. It is an exter nal test, and thu s it cannot access isSpace direc tly, so fmt op ens a back door to it by declaring an exp orted var iable that holds the int ernal isSpace func tion. This is the ent irety of the fmt package’s export_test.go file.

package fmt

var IsSpace = isSpace

This test file defines no tests; it just declares the exp orted symbol fmt.IsSpace for use by the exter nal test. This trick can also be used whene ver an exter nal test needs to use some of the te chniques of white-b ox testing .

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

316

CHAPTER 11. TESTING

11.2.5. Writing Effective Tests

Many newcomers to Go are sur prised by the minimalism of Go’s testing framewor k. Other languages’ framewor ks provide mechanisms for identifying test functions (of ten using reflection or met adat a), ho oks for per for ming ‘‘setup’’ and ‘‘te ardow n’’ op erat ions before and after the tests run, and librar ies of utilit y func tions for ass erting common pre dic ates, comp aring values, for matting error messages, and aborting a fai le d test (of ten using exception s).

Although these mechanisms can make tests ver y concis e, the resulting tests often seem like they are writt en in a foreig n language . Furthermore, alt hough the y may rep ort PASS or FAIL

correc tly, their manner may be unf riendly to the unfor tunate maintainer, wit h cr yptic fai lure mess ages like "assert: 0 == 1" or page after page of stack traces.

Go’s att itude to testing stand s in stark contrast. It exp ects test aut hor s to do most of this wor k themselves, defining functions to avoid rep etit ion, just as the y would for ordinar y prog rams.

The pro cess of testing is not one of rot e form filling; a test has a user int erface too, alb eit one whos e on ly users are als o its maintainers. A go o d test does not explo de on fai lure but prints a clear and succinct des crip tion of the sympt om of the pro blem, and perhaps other relevant fac ts ab out the context. Ide ally, the maintainer should not need to read the source code to de ciph er a test fai lure. A go o d test should not give up after one fai lure but should try to rep ort several erro rs in a single run, since the pattern of fai lures may its elf be revealing.

The ass ertion function below compares two values, cons tructs a gener ic er ror message , and ptg16091132

stops the program. It’s easy to use and it’s cor rec t, but when it fai ls, the error message is almost us eless. It does not solve the hard pro blem of providing a good user int erface.

import (

"fmt"

"strings"

"testing"

)

// A poor assertion function.

func assertEqual(x, y int) {

if x != y {

panic(fmt.Sprintf("%d != %d", x, y))

}

}

func TestSplit(t *testing.T) {

words := strings.Split("a:b:c", ":") assertEqual(len(words), 3)

// ...

}

In this sense, ass ertion functions suf fer fro m premat ure abstrac tion: by tre ating the fai lure of this par tic ular test as a mere dif ference of two int egers, we for feit the opp ortunity to provide me aningf ul cont ext. We can provide a better message by starting fro m the con crete det ails, as in the example below. Only once rep etit ive patterns emerge in a given test suite is it time to introduce abstrac tions.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 11.2. TEST FUNCTIONS

317

func TestSplit(t *testing.T) {

s, sep := "a:b:c", ":"

words := strings.Split(s, sep)

if got, want := len(words), 3; got != want {

t.Errorf("Split(%q, %q) returned %d words, want %d", s, sep, got, want)

}

// ...

}

No w the test rep orts the function that was cal le d, its inputs, and the sig nificance of the result; it explicitly identifies the actual value and the exp ectat ion; and it continues to exe cut e even if this ass ertion should fai l. Once we’ve writt en a test like this, the natural next step is often not to define a function to replace the ent ire if st atement, but to exe cut e the test in a loop in which s, sep, and want var y, like the table-dr iven test of IsPalindrome.

The pre vious example didn’t need any utilit y func tions, but of cours e that shouldn’t stop us from int roducing functions when the y help make the code simpler. (We’ll look at one such ut i lit y func tion, reflect.DeepEqual, in Sec tion 13.3.) The key to a good test is to start by implementing the con crete beh avior that you want and only then use functions to simplif y the co de and eliminate rep etit ion. Best results are rarely obt ained by starting wit h a librar y of abstrac t, generic testing functions.

Exercis e 11.5: Extend TestSplit to use a table of inp uts and exp ected out puts.

ptg16091132

11.2.6. Avoiding Brittle Tests

An app lic ation that often fails when it encount ers new but valid inputs is cal le d bug gy; a test that spuriou sly fai ls when a sound change was made to the program is cal le d br ittle. Just as a bugg y prog ram frust rates its users, a britt le test exasperates its maintainers. The most britt le tests, which fai l for almost any change to the pro duc tion code, good or bad, are som etimes called ch ange detec tor or statu s qu o tests, and the time spent dealing wit h them can quickly deplete any benefit the y on ce seeme d to provide.

Wh en a function under test pro duces a comp lex out put such as a lon g st ring, an elaborate dat a st ruc ture, or a file, it’s tempt ing to che ck that the out put is exac tly equ al to som e ‘‘go lden’’ value that was expec ted when the test was writt en. But as the program evo l ves, par ts of the out put wi l l li kely change , prob ably in good ways, but change non etheless. And it’s not just the out put; func tions wit h comp lex inputs often bre ak because the inp ut used in a test is no lon g er valid.

The easiest way to avoid britt le tests is to che ck only the pro per ties you care about. Test your prog ram’s simpler and more stable int erfaces in preference to its internal functions. Be selective in your ass ertions. Don’t che ck for exac t st ring match es, for example, but look for relevant su bst rings that will remain unch ange d as the program evo l ves. It’s often wor th writing a su bst ant ial function to distill a complex out put dow n to its essence so that ass ertions will be re liable. Even thoug h that may seem like a lot of up-f ront effor t, it can pay for itself quickly in time that wou ld other wis e be spent fixing spuriou sly fai ling tests.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

318

CHAPTER 11. TESTING

11.3. Coverage

By its nature, test ing is never comp lete. As the influential comp uter scientist Edsger Dijkstra put it, ‘‘Test ing shows the presence, not the abs ence of bugs.’’ No quant ity of tests can ever prov e a package fre e of bugs. At best, the y increase our confidence that the package wor ks we ll in a wide range of imp ortant scenar ios.

The deg ree to which a test suite exercis es the package under test is cal le d the test’s covera ge.

Coverage can’t be quant ified direc tly—the dynamics of all but the most trivial programs are beyond pre cis e me asurement—but there are heur ist ics that can help us direc t our testing ef for ts to where the y are more likely to be useful.

St atement cov era ge is the simplest and most widely used of these heur ist ics. The statement coverage of a test suite is the frac tion of source statements that are exe cut ed at least once during the test. In this sec tion, we’ll use Go’s cover to ol, which is int egrated int o go test, to me asure statement cov erage and help identify obv iou s gaps in the tests.

The code below is a table-dr iven test for the expression evaluator we bui lt back in Chapt er 7: gopl.io/ch7/eval

func TestCoverage(t *testing.T) {

ptg16091132

var tests = []struct {

input string

env

Env

want string // expected error from Parse/Check or result from Eval

}{

{"x % 2", nil, "unexpected '%'"},

{"!true", nil, "unexpected '!'"},

{"log(10)", nil, ùnknown function "log"`},

{"sqrt(1, 2)", nil, "call to sqrt has 2 args, want 1"},

{"sqrt(A / pi)", Env{"A": 87616, "pi": math.Pi}, "167"},

{"pow(x, 3) + pow(y, 3)", Env{"x": 9, "y": 10}, "1729"},

{"5 / 9 * (F - 32)", Env{"F": -40}, "-40"},

}

for _, test := range tests {

expr, err := Parse(test.input)

if err == nil {

err = expr.Check(map[Var]bool{})

}

if err != nil {

if err.Error() != test.want {

t.Errorf("%s: got %q, want %q", test.input, err, test.want)

}

continue

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 11.3. COVERAGE

319

got := fmt.Sprintf("%.6g", expr.Eval(test.env)) if got != test.want {

t.Errorf("%s: %v => %s, want %s",

test.input, test.env, got, test.want)

}

}

}

First, let’s che ck that the test passes:

$ go test -v -run=Coverage gopl.io/ch7/eval

=== RUN TestCoverage

--- PASS: TestCoverage (0.00s)

PASS

ok

gopl.io/ch7/eval

0.011s

This command displ ays the usage message of the cov erage tool: $ go tool cover

Usage of 'go tool cover':

Given a coverage profile produced by 'go test':

go test -coverprofile=c.out

Open a web browser displaying annotated source code:

go tool cover -html=c.out

...

ptg16091132

The go tool command runs one of the exe cut ables fro m the Go toolchain. Thes e prog rams live in the direc tor y $GOROOT/pkg/tool/${GOOS}_${GOARCH}. Thanks to go build, we rarely ne e d to inv oke them direc tly.

No w we run the test wit h the -coverprofile flag:

$ go test -run=Coverage -coverprofile=c.out gopl.io/ch7/eval ok

gopl.io/ch7/eval

0.032s coverage: 68.5% of statements

This flag enables the col lec tion of cov erage dat a by inst rum ent ing the pro duc tion code. That is, it modifies a copy of the source code so that before each blo ck of statements is exe cut ed, a boole an var iable is set, wit h on e var iable per blo ck. Just before the modified program exits, it wr ites the value of each var iable to the specified log file c.out and prints a summar y of the frac tion of statements that were exe cut ed. (If all you need is the summar y, use go test -cover.)

If go test is run wit h the -covermode=count flag , the ins trument ation for each blo ck increments a count er ins tead of setting a boole an. The resulting log of exe cut ion counts of each block enables quant itative comparisons bet ween ‘‘hott er’’ blocks, which are more fre quently exec ute d, and ‘‘colder’’ on es.

Having gat hered the dat a, we run the cover to ol, which pro cesses the log , generates an HTML

report, and opens it in a new brows er window (Figure 11.3).

$ go tool cover -html=c.out

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

320

CHAPTER 11. TESTING

ptg16091132

Figure 11.3. A cov erage rep ort.

Each statement is colored green if it was covere d or red if it was not cov ere d. For clarity, we’ve sh ade d the backg round of the red text. We can see immediate ly that non e of our inputs exercised the unary operator Eval method. If we add this new test cas e to the table and re-r un the previous two command s, the unary expression code becom es gre en:

{"-x * -x", eval.Env{"x": 2}, "4"}

The two panic st atements remain red, how ever. This should not be sur prising, because these st atements are sup pos ed to be unreach able.

Ac hie ving 100% statement cov erage sound s li ke a noble goal, but it is not usu ally feasible in prac tice, nor is it likely to be a good use of effor t. Just because a statement is exe cut ed does not mean it is bug-f ree; statements containing complex expressions must be exe cut ed many times wit h dif ferent inp uts to cov er the int erest ing cas es. Some statements, like the panic st atements above , can never be reach ed. Others, such as those that handle esoteric error s, are hard to exercis e but rarely reach ed in prac tice. Testing is fundament ally a prag mat ic ende avor, a trade-off bet ween the cost of writing tests and the cost of fai lures that cou ld have been prevente d by tests. Cov erage tools can help identify the weakest spots, but devising good test cases demands the same rigorou s thin king as programming in general.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 11.4. BENCHMARK FUNCTIONS

321

11.4. Benchmark Functions

Benchmarking is the prac tice of measuring the per for mance of a program on a fixe d work load. In Go, a benchmark function looks like a test function, but wit h the Benchmark prefix and a *testing.B parameter that provides most of the same met hods as a *testing.T, plus a few ext ra re lated to per for mance measurement. It als o exp oses an integer field N, which sp ecifies the number of times to per for m the operat ion being measure d.

Here’s a benchmark for IsPalindrome that cal ls it N times in a loop.

import "testing"

func BenchmarkIsPalindrome(b *testing.B) {

for i := 0; i < b.N; i++ {

IsPalindrome("A man, a plan, a canal: Panama")

}

}

We run it wit h the command below. Unlike tests, by defau lt no benchmarks are run. The argument to the -bench flag selec ts which benchmarks to run. It is a regu lar expression matching the names of Benchmark func tions, wit h a defau lt value that match es non e of them.

The ‘‘.’’ patt ern causes it to match all benchmarks in the word package, but since there’s only ptg16091132

on e, -bench=IsPalindrome would have been equivalent.

$ cd $GOPATH/src/gopl.io/ch11/word2

$ go test -bench=.

PASS

BenchmarkIsPalindrome-8 1000000

1035 ns/op

ok

gopl.io/ch11/word2

2.179s

The benchmark name’s numer ic suffix, 8 here , indic ates the value of GOMAXPROCS, which is important for con cur rent benchmarks.

The rep ort tel ls us that each cal l to IsPalindrome to ok about 1.035 micros econd s, averaged ov er 1,000,000 runs. Since the benchmark runner initial ly has no ide a how lon g the operat ion takes, it makes som e initial measurements using small values of N and then ext rapol ates to a value large enoug h for a stable timing measurement to be made.

The reason the loop is imp lemente d by the benchmark function, and not by the cal ling code in the test driver, is so that the benchmark function has the opp ortunity to exe cut e any necessary on e-t ime setup code outside the loop wit hout this adding to the measured time of each iteration. If this setup code is still per turbing the results, the testing.B parameter provides met hods to stop, resume, and res et the timer, but these are rarely needed.

No w that we have a benchmark and tests, it’s easy to try out ide as for mak ing the program faster. Perhaps the most obviou s opt imizat ion is to make IsPalindrome’s secon d lo op stop ch eck ing at the midpoint, to avoid doing each comparison twice:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

322

CHAPTER 11. TESTING

n := len(letters)/2

for i := 0; i < n; i++ {

if letters[i] != letters[len(letters)-1-i] {

return false

}

}

return true

But as is often the cas e, an obv iou s opt imizat ion doesn’t always yield the exp ected benefit.

This one delivered a mere 4% imp rov ement in one exp eriment.

$ go test -bench=.

PASS

BenchmarkIsPalindrome-8 1000000

992 ns/op

ok

gopl.io/ch11/word2

2.093s

Anot her idea is to pre-al locate a sufficiently large array for use by letters, rat her than exp and it by successive cal ls to append. Declaring letters as an array of the rig ht size, like this,

letters := make([]rune, 0, len(s))

for _, r := range s {

if unicode.IsLetter(r) {

letters = append(letters, unicode.ToLower(r))

}

ptg16091132

}

yields an imp rov ement of nearly 35%, and the benchmark runner now rep orts the average ov er 2,000,000 iterat ions.

$ go test -bench=.

PASS

BenchmarkIsPalindrome-8 2000000

697 ns/op

ok

gopl.io/ch11/word2

1.468s

As this example shows, the fastest program is often the one that makes the fewest memor y al location s. The -benchmem command-line flag will include memor y al location statist ics in its report. Here we compare the number of allocat ions before the opt imizat ion: $ go test -bench=. -benchmem

PASS

BenchmarkIsPalindrome

1000000

1026 ns/op

304 B/op

4 allocs/op

and after it:

$ go test -bench=. -benchmem

PASS

BenchmarkIsPalindrome

2000000

807 ns/op 128 B/op 1 allocs/op

Cons olid ating the allocat ions in a single cal l to make eliminated 75% of the allocat ions and halved the quant ity of allocated memor y.

Benchmarks like this tel l us the abs olut e time required for a given operat ion, but in many settings the int erest ing per for mance question s are about the re lat ive timings of two dif ferent

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 11.5. PROFILING

323

op erat ions. For example, if a function takes 1ms to pro cess 1,000 elements, how lon g wi l l it take to pro cess 10,000 or a million? Such comparisons reveal the asy mpt otic growt h of the running time of the function. Another example: what is the best size for an I/O buf fer?

Benchmarks of app lic ation throughput over a range of sizes can help us cho ose the smallest buffer that delivers sat isfac tor y perfor mance. A third example: which algor it hm per for ms best for a given job? Benchmarks that evaluate two dif ferent algor it hms on the same inp ut dat a can often show the strengt hs and weaknesses of each one on imp ortant or represent ative work loads.

Comp arat ive benchmarks are just regu lar code. The y typically take the for m of a single parameter ized function, cal le d from several Benchmark func tions wit h dif ferent values, like this:

func benchmark(b *testing.B, size int) { /* ... */ }

func Benchmark10(b *testing.B)

{ benchmark(b, 10) }

func Benchmark100(b *testing.B)

{ benchmark(b, 100) }

func Benchmark1000(b *testing.B) { benchmark(b, 1000) }

The parameter size, which specifies the size of the inp ut, varies across benchmarks but is cons tant wit hin each benchmark. Resist the tempt ation to use the parameter b.N as the inp ut size. Unless you int erpret it as an iterat ion count for a fixe d-size input, the results of your benchmark will be meaningless.

Patt erns reveale d by comparat ive benchmarks are par tic ularly useful dur ing program design, ptg16091132

but we don’t throw the benchmarks away when the program is wor king. As the program evolves, or its input grows, or it is deploye d on new operat ing systems or pro cessors wit h different charac ter ist ics, we can reuse those benchmarks to revisit desig n de cisions.

Exercis e 11.6: Wr ite benchmarks to compare the PopCount implementation in Sec tion 2.6.2

with your solut ion s to Exercis e 2.4 and Exercis e 2.5. At what point does the table-b ased approach bre ak even?

Exercis e 11.7: Wr ite benchmarks for Add, UnionWith, and other met hods of *IntSet (§6.5) using large pseudo-random inp uts. How fast can you make these met hods run? How does the ch oice of word size affe ct per for mance? How fast is IntSet comp are d to a set implementation based on the bui lt-in map typ e?

11.5. Pro filing

Benchmarks are useful for measuring the per for mance of specific operat ions, but when we’re tr ying to make a slow program faster, we often have no ide a where to beg in. Ev ery programmer knows Don ald Knu th’s aph orism about premature opt imizat ion, which app eared in

‘‘St ruc tured Programming wit h go to Statements’’ in 1974. Although often misinterprete d to me an per for mance doesn’t matter, in its original context we can discer n a dif ferent meaning: There is no dou bt that the grai l of efficiency leads to abu se. Programmers waste enor mou s amounts of time thin king about, or wor rying about, the speed of non critical

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

324

CHAPTER 11. TESTING

parts of their programs, and these att emp ts at efficiency actually have a stron g negat ive impact when debugging and maintenance are con sidered. We sh oul d forget about smal l efficiencies, say about 97% of the time: premature opt imizat ion is the root of all evil.

Yet we should not pass up our opp ortunities in that critical 3%. A go o d prog rammer wi l l not be lulled int o comp lacency by such reasoning, he will be wis e to look caref ully at the critical code; but only af ter that code has been identified. It is often a mist ake to make a prior i judg ments about what par ts of a prog ram are really critical, since the univers al exp erience of programmers who have been using measurement tools has been that their intuit ive guesses fai l.

Wh en we wish to look caref ully at the speed of our programs, the best technique for identifying the critical code is profiling. Profiling is an aut omated appro ach to per for mance measurement bas ed on sampling a number of profile events during exe cut ion, then ext rapol ating fro m them dur ing a post-pro cessing step; the resulting statist ical summar y is cal le d a profile.

Go sup por ts many kinds of profiling, each con cer ned wit h a dif ferent asp ect of per for mance, but all of them inv olve recording a sequence of events of int erest, each of which has an accom-pany ing stack trace—the stack of function cal ls active at the moment of the event. The go test to ol has bui lt-in sup por t for several kinds of profiling.

ptg16091132

A CPU profile identifies the functions whose exe cut ion requires the most CPU time. The current ly running thread on each CPU is int err upt ed per iodic ally by the operat ing system every fe w mi l lisecond s, wit h each int err upt ion recording one profile event before nor mal exec ution resumes.

A he ap profile identifies the statements responsible for allocat ing the most memor y. The profiling librar y samp les cal ls to the int ernal memor y al location routines so that on average, on e profile event is recorde d per 512KB of allocated memor y.

A bl o cki n g profile identifies the operat ions responsible for blo cking goro utines the lon g est, such as system cal ls, channel sends and receives, and acquisition s of locks. The profiling librar y re cords an event every time a goroutine is blo cke d by one of these operat ions.

Gat her ing a profile for code under test is as easy as enabling one of the flags below. Be caref ul when using more than one flag at a time, how ever: the machiner y for gat her ing one kind of profile may ske w the results of others.

$ go test -cpuprofile=cpu.out

$ go test -blockprofile=block.out

$ go test -memprofile=mem.out

It’s easy to add profiling sup por t to non-test programs too, thoug h the det ails of how we do that vary bet ween short-live d command-line tools and lon g-r unning ser ver applic ations.

Profiling is esp eci ally useful in lon g-r unning app lic ations, so the Go runtime’s profiling fe atures can be enabled under programmer cont rol using the runtime API.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 11.5. PROFILING

325

Once we’ve gat hered a profile, we need to analyze it using the pprof to ol. This is a stand ard part of the Go distr ibution, but since it’s not an everyday tool, it’s accessed indirec tly using go tool pprof. It has dozens of features and opt ion s, but basic use requires only two arguments, the exe cut able that pro duce d the profile and the profile log.

To make profiling efficient and to save space, the log does not include function names; instead, func tions are identified by their addresses. This means that pprof ne e ds the exe cut able in order to make sense of the log . Although go test usually dis cards the test exe cut able once the test is complete, when profiling is enabled it saves the exe cut able as foo.test, where foo is the name of the teste d package.

The command s below show how to gat her and displ ay a simple CPU profile. We’ve selec ted on e of the benchmarks fro m the net/http package. It is usu ally better to profile specific benchmarks that have been con str ucted to be represent ative of wor klo ads one cares about.

Benchmarking test cas es is almost never represent ative , which is why we dis abled them by using the filt er -run=NONE.

$ go test -run=NONE -bench=ClientServerParallelTLS64 \

-cpuprofile=cpu.log net/http

PASS

BenchmarkClientServerParallelTLS64-8 1000

3141325 ns/op

143010 B/op

1747 allocs/op

ok

net/http

3.395s

ptg16091132

$ go tool pprof -text -nodecount=10 ./http.test cpu.log 2570ms of 3590ms total (71.59%)

Dropped 129 nodes (cum <= 17.95ms)

Showing top 10 nodes out of 166 (cum >= 60ms)

flat flat%

sum%

cum

cum%

1730ms 48.19% 48.19%

1750ms 48.75%

crypto/elliptic.p256ReduceDegree

230ms 6.41% 54.60%

250ms

6.96% crypto/elliptic.p256Diff

120ms 3.34% 57.94%

120ms

3.34% math/big.addMulVVW

110ms 3.06% 61.00%

110ms

3.06% syscall.Syscall

90ms 2.51% 63.51% 1130ms 31.48% crypto/elliptic.p256Square 70ms 1.95% 65.46%

120ms

3.34% runtime.scanobject

60ms 1.67% 67.13%

830ms 23.12% crypto/elliptic.p256Mul

60ms 1.67% 68.80%

190ms

5.29% math/big.nat.montgomery

50ms 1.39% 70.19%

50ms

1.39% crypto/elliptic.p256ReduceCarry

50ms 1.39% 71.59%

60ms

1.67% crypto/elliptic.p256Sum

The -text flag specifies the out put for mat, in this cas e, a textual table wit h on e row per function, sor ted so the ‘‘hott est’’ func tions—thos e that cons ume the most CPU cyc les—appear firs t. The -nodecount=10 flag limits the result to 10 rows. For gross per for mance pro blems, this textual for mat may be enoug h to pinpoint the cause.

This profile tel ls us that ellip tic-c urve crypt ography is imp ortant to the per for mance of this partic ular HTTPS benchmark. By contrast, if a profile is dominated by memor y al location func tions fro m the runtime package, reducing memor y cons ump tion may be a wor thw hile opt imizat ion.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

326

CHAPTER 11. TESTING

Fo r more subtle pro blems, you may be better off using one of pprof’s graphic al displays.

Thes e re quire GraphViz, which can be dow nlo ade d from www.graphviz.org. The -web flag then renders a direc ted graph of the functions of the program, annot ate d by their CPU profile numb ers and colored to indic ate the hott est functions.

We’ve only scratch ed the sur face of Go’s profiling tools here. To find out more, read the

‘‘Profiling Go Programs’’ ar ticle on the Go Blog .

11.6. Example Functions

The third kind of function tre ate d sp eci ally by go test is an example function, one whose name starts wit h Example. It has neither parameters nor results. Here’s an example function for IsPalindrome:

func ExampleIsPalindrome() {

fmt.Println(IsPalindrome("A man, a plan, a canal: Panama")) fmt.Println(IsPalindrome("palindrome"))

// Output:

// true

// false

}

ptg16091132

Example functions ser ve three pur pos es. The primary one is documentation: a good example can be a more succinct or int uit ive way to conve y the beh avior of a librar y func tion than its pros e des crip tion, especi ally when used as a reminder or quick reference. An example can also demon strate the int erac tion bet ween several typ es and functions belon ging to one API, whereas prose documentation must always be att ach ed to one place, like a typ e or function de clarat ion or the package as a whole. And unlike examples wit hin comments, example function s are real Go code, subjec t to compi le-t ime che cking , so the y don’t becom e st ale as the co de evolves.

Based on the suffix of the Example func tion, the web-b ased documentation ser ver godoc asso ciates example functions wit h the function or package the y exemplif y, so ExampleIsPalindrome would be shown with the documentation for the IsPalindrome func tion, and an example function cal le d ju st Example would be ass oci ated wit h the word package as a whole.

The secon d purpos e is that examples are exe cut able tests run by go test. If the example function contains a final // Output: comment like the one above , the test driver will exe cut e the func tion and che ck that what it print ed to its stand ard out put match es the text wit hin the comment.

The third pur pos e of an example is hands-on exp erimentation. The godoc server at golang.org us es the Go Playg round to let the user edit and run each example function fro m within a web brows er, as shown in Figure 11.4. This is often the fastest way to get a feel for a partic ular function or langu age feature .

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 11.6. EXAMPLE FUNCTIONS

327

ptg16091132

Figure 11.4. An int erac tive example of strings.Join in godoc.

The final two chapt ers of the book examine the reflect and unsafe packages, which few Go prog rammers regu larly use—and even fewer ne e d to use. If you haven’t writt en any subst ant ial Go programs yet, now wou ld be a good time to do that.

This page intentionally left blank ptg16091132

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

12

Re flection

Go provides a mechanism to update var iables and ins pec t their values at run time, to cal l their methods, and to app l y the operat ions int rinsic to their represent ation, all wit hout knowing their typ es at comp ile time. This mechanism is cal le d reflec tion. Reflec tion als o lets us tre at types thems elves as firs t-class values.

ptg16091132

In this chapt er, we’ll explore Go’s reflec tion features to see how the y increase the expressiveness of the langu age, and in par tic ular how the y are cruci al to the imp lementation of two important APIs: str ing for matting provide d by fmt, and pro tocol encoding provide d by packages like encoding/json and encoding/xml. Reflec tion is als o essent ial to the templ ate me chanism provide d by the text/template and html/template packages we saw in Section 4.6. How ever, reflec tion is complex to reason about and not for casual use, so although these packages are imp lemente d using reflec tion, the y do not exp ose reflec tion in their own APIs.

12.1. Why Reflection?

Sometimes we need to write a function cap able of dealing unifor mly wit h values of typ es that don’t sat isf y a common int erface, don’t have a known represent ation, or don’t exist at the time we desig n the function—or even all three.

A fami liar example is the for matting log ic within fmt.Fprintf, which can usefully print an arbit rar y value of any typ e, even a user-define d on e. Let’s try to imp lement a function like it using what we know already. For simplicity, our function will accept one argument and will return the result as a str ing like fmt.Sprint do es, so we’ll cal l it Sprint.

We start wit h a typ e sw itch that tests whether the argument defines a String method, and cal l it if so. We then add swit c h cases that test the value’s dynamic typ e against each of the basic 329

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

330

CHAPTER 12. REFLECTION

types—string, int, bool, and so on—and per for m the appro priate for matting operat ion in each cas e.

func Sprint(x interface{}) string {

type stringer interface {

String() string

}

switch x := x.(type) {

case stringer:

return x.String()

case string:

return x

case int:

return strconv.Itoa(x)

// ...similar cases for int16, uint32, and so on...

case bool:

if x {

return "true"

}

return "false"

default:

// array, chan, func, map, pointer, slice, struct

return "???"

}

ptg16091132

}

But how do we deal wit h ot her typ es, li ke []float64, map[string][]string, and so on? We could add more cas es, but the number of such typ es is infinite. And what about named typ es, li ke url.Values?

Even if the typ e sw itch had a cas e for its underly ing typ e map[string][]string, it wou ldn’t match url.Values becaus e the two typ es are not identical, and the typ e sw itch cannot include a cas e for each typ e li ke url.Values becaus e that would require this librar y to dep end upon its clients.

Wi thout a way to ins pec t the represent ation of values of unknown types, we quickly get stuck.

Wh at we need is reflec tion.

12.2. reflect.Type and reflect.Value

Reflec tion is provide d by the reflect package. It defines two imp ortant typ es, Type and Value. A Type repres ents a Go typ e. It is an int erface wit h many met hods for dis criminat ing amon g types and ins pec ting their comp onents, like the fields of a str uct or the parameters of a func tion. The sole imp lementation of reflect.Type is the typ e des crip tor (§7.5), the same entity that identifies the dynamic typ e of an int erface value.

The reflect.TypeOf func tion accepts any interface{} and retur ns its dynamic typ e as a reflect.Type:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 12.2. REFLECT.TYPE AND REFLECT.V ALUE

331

t := reflect.TypeOf(3) // a reflect.Type

fmt.Println(t.String()) // "int"

fmt.Println(t)

// "int"

The TypeOf(3) call above assig ns the value 3 to the interface{} parameter. Recal l from Section 7.5 that an assig nment fro m a con crete value to an int erface typ e perfor ms an imp licit interface conv ersion, which cre ates an int erface value consisting of two components: its dy namic type is the operand’s typ e (int) and its dy namic valu e is the operand’s value (3).

Becaus e reflect.TypeOf returns an int erface value’s dynamic typ e, it always retur ns a concrete typ e. So, for example, the code below prints "*os.File", not "io.Writer". Later, we wi l l see that reflect.Type is cap able of represent ing int erface typ es to o.

var w io.Writer = os.Stdout

fmt.Println(reflect.TypeOf(w)) // "*os.File"

No tice that reflect.Type satisfies fmt.Stringer. Because print ing the dynamic typ e of an interface value is useful for debug ging and log ging, fmt.Printf prov ides a shorthand, %T, that us es reflect.TypeOf internal ly:

fmt.Printf("%T\n", 3) // "int"

The other important typ e in the reflect package is Value. A reflect.Value can hold a value of any typ e. The reflect.ValueOf func tion accepts any interface{} and retur ns a reflect.Value cont aining the int erface’s dynamic value. As wit h reflect.TypeOf, the ptg16091132

resu lts of reflect.ValueOf are always concrete, but a reflect.Value can hold int erface values too.

v := reflect.ValueOf(3) // a reflect.Value

fmt.Println(v)

// "3"

fmt.Printf("%v\n", v)

// "3"

fmt.Println(v.String()) // NOTE: "<int Value>"

Like reflect.Type, reflect.Value also sat isfies fmt.Stringer, but unless the Value holds a str ing , the result of the String method reveals only the typ e. Ins tead, use the fmt package’s

%v verb, which tre ats reflect.Values speci ally.

Calling the Type method on a Value returns its typ e as a reflect.Type: t := v.Type()

// a reflect.Type

fmt.Println(t.String()) // "int"

The inv ers e op erat ion to reflect.ValueOf is the reflect.Value.Interface method. It returns an interface{} holding the same con crete value as the reflect.Value: v := reflect.ValueOf(3) // a reflect.Value

x := v.Interface()

// an interface{}

i := x.(int)

// an int

fmt.Printf("%d\n", i)

// "3"

A reflect.Value and an interface{} can bot h hold arbit rar y values. The dif ference is that an empt y interface hides the represent ation and int rinsic operat ions of the value it holds and exp oses non e of its met hods, so unless we know its dynamic typ e and use a typ e assertion to

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

332

CHAPTER 12. REFLECTION

peer inside it (as we did abov e), there is lit tle we can do to the value wit hin. In contrast, a Value has many met hods for ins pec ting its cont ents, regardless of its typ e. Let’s use them for our secon d attemp t at a general for matting function, which we’ll cal l format.Any.

In ste ad of a typ e sw itch, we use reflect.Value’s Kind method to dis criminate the cas es.

Although there are infinite ly many typ es, there are only a finite number of ki nds of typ e: the basic typ es Bool, String, and all the numbers; the aggregate typ es Array and Struct; the reference typ es Chan, Func, Ptr, Slice, and Map; Interface types; and finally Invalid, meaning no value at all. (The zero value of a reflect.Value has kind Invalid.) gopl.io/ch12/format

package format

import (

"reflect"

"strconv"

)

// Any formats any value as a string.

func Any(value interface{}) string {

return formatAtom(reflect.ValueOf(value))

}

// formatAtom formats a value without inspecting its internal structure.

func formatAtom(v reflect.Value) string {

ptg16091132

switch v.Kind() {

case reflect.Invalid:

return "invalid"

case reflect.Int, reflect.Int8, reflect.Int16,

reflect.Int32, reflect.Int64:

return strconv.FormatInt(v.Int(), 10)

case reflect.Uint, reflect.Uint8, reflect.Uint16,

reflect.Uint32, reflect.Uint64, reflect.Uintptr:

return strconv.FormatUint(v.Uint(), 10)

// ...floating-point and complex cases omitted for brevity...

case reflect.Bool:

return strconv.FormatBool(v.Bool())

case reflect.String:

return strconv.Quote(v.String())

case reflect.Chan, reflect.Func, reflect.Ptr, reflect.Slice, reflect.Map: return v.Type().String() + " 0x" +

strconv.FormatUint(uint64(v.Pointer()), 16)

default: // reflect.Array, reflect.Struct, reflect.Interface return v.Type().String() + " value"

}

}

So far, our function tre ats each value as an indivisible thing wit h no int ernal str ucture—hence formatAtom. For aggregate typ es (str ucts and arrays) and int erfaces it prints only the ty pe of the value, and for reference typ es (channel s, functions, point ers, slices, and maps), it prints the type and the reference address in hexade cimal. This is less than ide al but still a maj or

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 12.3. DISPLAY, A RECURSIVE VALUE PRINTER

333

improv ement, and since Kind is con cer ned only wit h the underly ing represent ation, format.Any works for named typ es to o. For example:

var x int64 = 1

var d time.Duration = 1 * time.Nanosecond

fmt.Println(format.Any(x))

// "1"

fmt.Println(format.Any(d))

// "1"

fmt.Println(format.Any([]int64{x}))

// "[]int64 0x8202b87b0"

fmt.Println(format.Any([]time.Duration{d})) // "[]time.Duration 0x8202b87e0"

12.3. Display, a Recursive Value Printer

Next we’ll take a look at how to imp rov e the displ ay of composite typ es. Rather than try to copy fmt.Sprint exac tly, we’ll bui ld a debug ging utilit y func tion cal le d Display that, given an arbit rar ily complex value x, prints the complete str ucture of that value, lab eling each element wit h the pat h by which it was found. Let’s start wit h an example.

e, _ := eval.Parse("sqrt(A / pi)")

Display("e", e)

In the cal l ab ove , the argument to Display is a syntax tre e from the expression evaluator in Section 7.9. The out put of Display is shown below :

ptg16091132

Display e (eval.call):

e.fn = "sqrt"

e.args[0].type = eval.binary

e.args[0].value.op = 47

e.args[0].value.x.type = eval.Var

e.args[0].value.x.value = "A"

e.args[0].value.y.type = eval.Var

e.args[0].value.y.value = "pi"

Wh ere possible, you should avoid exposing reflec tion in the API of a package. We’ll define an unexp orted function display to do the real wor k of the rec ursion, and exp ort Display, a simple wrapper aro und it that accepts an interface{} parameter : gopl.io/ch12/display

func Display(name string, x interface{}) {

fmt.Printf("Display %s (%T):\n", name, x)

display(name, reflect.ValueOf(x))

}

In display, we’ll use the formatAtom func tion we define d earlier to print elementary values—

basic typ es, func tions, and channel s—but we’ll use the methods of reflect.Value to rec ursive ly displ ay each component of a more complex typ e. As the rec ursion des cends, the path st ring, which initial ly des crib es the starting value (for ins tance, "e"), will be aug mente d to indic ate how we reach ed the cur rent value (for ins tance, "e.args[0].value").

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

334

CHAPTER 12. REFLECTION

Since we’re no lon g er pretending to imp lement fmt.Sprint, we will use the fmt package to ke ep our example short.

func display(path string, v reflect.Value) {

switch v.Kind() {

case reflect.Invalid:

fmt.Printf("%s = invalid\n", path)

case reflect.Slice, reflect.Array:

for i := 0; i < v.Len(); i++ {

display(fmt.Sprintf("%s[%d]", path, i), v.Index(i))

}

case reflect.Struct:

for i := 0; i < v.NumField(); i++ {

fieldPath := fmt.Sprintf("%s.%s", path, v.Type().Field(i).Name) display(fieldPath, v.Field(i))

}

case reflect.Map:

for _, key := range v.MapKeys() {

display(fmt.Sprintf("%s[%s]", path,

formatAtom(key)), v.MapIndex(key))

}

case reflect.Ptr:

if v.IsNil() {

fmt.Printf("%s = nil\n", path)

ptg16091132

} else {

display(fmt.Sprintf("(*%s)", path), v.Elem())

}

case reflect.Interface:

if v.IsNil() {

fmt.Printf("%s = nil\n", path)

} else {

fmt.Printf("%s.type = %s\n", path, v.Elem().Type()) display(path+".value", v.Elem())

}

default: // basic types, channels, funcs

fmt.Printf("%s = %s\n", path, formatAtom(v))

}

}

Let’s dis cuss the cas es in order.

Slices and array s: The log ic is the same for bot h. The Len method retur ns the number of elements of a slice or array value, and Index(i) retr ieves the element at index i, als o as a reflect.Value; it panics if i is out of bound s. These are analogou s to the bui lt-in len(a) and a[i] op erat ions on sequences. The display func tion rec ursively inv okes its elf on each element of the sequence, app ending the subscript not ation "[i]" to the pat h.

Although reflect.Value has many met hods, only a few are safe to cal l on any given value.

Fo r example, the Index method may be cal le d on values of kind Slice, Array, or String, but panics for any other kind.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 12.3. DISPLAY, A RECURSIVE VALUE PRINTER

335

Structs: The NumField method rep orts the number of fields in the str uct, and Field(i) returns the value of the i-t h field as a reflect.Value. The list of fields includes ones prom ote d from anony mou s fields. To app end the field selec tor not ation ".f" to the pat h, we mu st obt ain the reflect.Type of the str uct and access the name of its i-t h field.

Ma ps: The MapKeys method retur ns a slice of reflect.Values, one per map key. As usu al when iterat ing over a map, the order is undefine d. MapIndex(key) returns the value corresp onding to key. We app end the subscript not ation "[key]" to the pat h. (We’re cut ting a corner here. The typ e of a map key isn’t restr icted to the typ es formatAtom hand les best; ar rays, st ruc ts, and int erfaces can also be valid map keys. Extending this cas e to print the key in full is Exercis e 12.1.)

Po int ers: The Elem method retur ns the var iable point ed to by a point er, again as a reflect.Value. This operat ion wou ld be safe even if the point er value is nil, in which cas e the result wou ld have kind Invalid, but we use IsNil to detec t ni l pointers explicitly so we can print a more appro priate message . We prefix the pat h with a "*" and parenthesize it to avoid ambiguity.

In ter fac es: Again, we use IsNil to test whether the int erface is nil, and if not, we ret rie ve its dy namic value using v.Elem() and print its typ e and value.

No w that our Display func tion is complete, let’s put it to wor k. The Movie type below is a ptg16091132

slig ht var iat ion on the one in Sec tion 4.5:

type Movie struct {

Title, Subtitle string

Year

int

Color

bool

Actor

map[string]string

Oscars

[]string

Sequel

*string

}

Let’s declare a value of this typ e and see what Display do es with it: strangelove := Movie{

Title:

"Dr. Strangelove",

Subtitle: "How I Learned to Stop Worrying and Love the Bomb", Year:

1964,

Color:

false,

Actor: map[string]string{

"Dr. Strangelove":

"Peter Sellers",

"Grp. Capt. Lionel Mandrake": "Peter Sellers",

"Pres. Merkin Muffley":

"Peter Sellers",

"Gen. Buck Turgidson":

"George C. Scott",

"Brig. Gen. Jack D. Ripper":

"Sterling Hayden",

`Maj. T.J. "King" Kong`:

"Slim Pickens",

},

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

336

CHAPTER 12. REFLECTION

Oscars: []string{

"Best Actor (Nomin.)",

"Best Adapted Screenplay (Nomin.)",

"Best Director (Nomin.)",

"Best Picture (Nomin.)",

},

}

The cal l Display("strangelove", strangelove) pr ints: Display strangelove (display.Movie):

strangelove.Title = "Dr. Strangelove"

strangelove.Subtitle = "How I Learned to Stop Worrying and Love the Bomb"

strangelove.Year = 1964

strangelove.Color = false

strangelove.Actor["Gen. Buck Turgidson"] = "George C. Scott"

strangelove.Actor["Brig. Gen. Jack D. Ripper"] = "Sterling Hayden"

strangelove.Actor["Maj. T.J. \"King\" Kong"] = "Slim Pickens"

strangelove.Actor["Dr. Strangelove"] = "Peter Sellers"

strangelove.Actor["Grp. Capt. Lionel Mandrake"] = "Peter Sellers"

strangelove.Actor["Pres. Merkin Muffley"] = "Peter Sellers"

strangelove.Oscars[0] = "Best Actor (Nomin.)"

strangelove.Oscars[1] = "Best Adapted Screenplay (Nomin.)"

strangelove.Oscars[2] = "Best Director (Nomin.)"

ptg16091132

strangelove.Oscars[3] = "Best Picture (Nomin.)"

strangelove.Sequel = nil

We can use Display to displ ay the int ernals of librar y types, such as *os.File: Display("os.Stderr", os.Stderr)

// Output:

// Display os.Stderr (*os.File):

// (*(*os.Stderr).file).fd = 2

// (*(*os.Stderr).file).name = "/dev/stderr"

// (*(*os.Stderr).file).nepipe = 0

No tice that even unexp orted fields are visible to reflec tion. Beware that the par tic ular out put of this example may vary across platfor ms and may change over time as librar ies evolve.

(Thos e fields are private for a reason!) We can even app l y Display to a reflect.Value and watch it traverse the int ernal represent ation of the typ e des crip tor for *os.File. The out put of the cal l Display("rV", reflect.ValueOf(os.Stderr)) is shown below, thoug h of cours e your mile age may vary:

Display rV (reflect.Value):

(*rV.typ).size = 8

(*rV.typ).hash = 871609668

(*rV.typ).align = 8

(*rV.typ).fieldAlign = 8

(*rV.typ).kind = 22

(*(*rV.typ).string) = "*os.File"

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 12.3. DISPLAY, A RECURSIVE VALUE PRINTER

337

(*(*(*rV.typ).uncommonType).methods[0].name) = "Chdir"

(*(*(*(*rV.typ).uncommonType).methods[0].mtyp).string) = "func() error"

(*(*(*(*rV.typ).uncommonType).methods[0].typ).string) = "func(*os.File) error"

...

Observe the dif ference bet ween these two examples:

var i interface{} = 3

Display("i", i)

// Output:

// Display i (int):

// i = 3

Display("&i", &i)

// Output:

// Display &i (*interface {}):

// (*&i).type = int

// (*&i).value = 3

In the firs t example, Display calls reflect.ValueOf(i), which retur ns a value of kind Int.

As we mention ed in Sec tion 12.2, reflect.ValueOf always retur ns a Value of a con crete typ e since it ext racts the contents of an int erface value.

In the secon d example, Display calls reflect.ValueOf(&i), which retur ns a point er to i, of kind Ptr. The switch cas e for Ptr calls Elem on this value, which retur ns a Value repres enting ptg16091132

the variab le i itself, of kind Interface. A Value obtained indirec tly, like this one, may represent any value at all, including int erfaces. The display func tion cal ls its elf rec ursively and this time, it prints sep arate components for the int erface’s dynamic typ e and value.

As cur rently imp lemente d, Display wi l l ne ver ter minate if it encount ers a cyc le in the obj e ct graph, such as this lin ked list that eats its own tai l:

// a struct that points to itself

type Cycle struct{ Value int; Tail *Cycle }

var c Cycle

c = Cycle{42, &c}

Display("c", c)

Display pr ints this ever-grow ing exp ansion:

Display c (display.Cycle):

c.Value = 42

(*c.Tail).Value = 42

(*(*c.Tail).Tail).Value = 42

(*(*(*c.Tail).Tail).Tail).Value = 42

...ad infinitum...

Many Go programs contain at least som e cyclic dat a. Making Display ro bust against such cycles is trick y, requir ing addition al bookkeeping to record the set of references that have been fo llowe d so far ; it is costly too. A general solut ion requires unsafe language features, as we wi l l see in Sec tion 13.3.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

338

CHAPTER 12. REFLECTION

Cy cles pos e less of a pro blem for fmt.Sprint becaus e it rarely tries to print the complete st ruc ture. For example, when it encount ers a point er, it bre aks the rec ursion by print ing the pointer’s numer ic value. It can get stuck trying to print a slice or map that contains its elf as an element, but such rare cas es do not war rant the con siderable extra tro uble of handling cyc les.

Exercis e 12.1: Extend Display so that it can display maps whose keys are str ucts or arrays.

Exercis e 12.2: Make display safe to use on cyc lic dat a st ruc tures by bounding the number of steps it takes before abandoning the rec ursion. (In Sec tion 13.3, we’ll see another way to detec t cycles.)

12.4. Example: Encoding S-Expressions

Display is a debug ging routine for displ aying str uctured dat a, but it’s not far short of being able to encode or marshal arbit rar y Go obj e cts as messages in a por table not ation suitable for inter-process communic ation.

As we saw in Sec tion 4.5, Go’s stand ard librar y supp orts a var iety of for mats, including JSON, XML, and ASN.1. Another not ation that is still widely used is S-expre ssi ons, the syntax of Lisp. Unlike the other not ation s, S-expressions are not sup por ted by the Go stand ard librar y, not least because the y have no univers ally accepte d definition, despit e several attemp ts at stan-ptg16091132

dardizat ion and the existence of many imp lementation s.

In this sec tion, we’ll define a package that encodes arbit rar y Go obj e cts using an S-expression notation that sup por ts the fol low ing con str ucts:

42

integer

"hello"

string (with Go-style quotation)

foo

symbol (an unquoted name)

(1 2 3)

list

(zero or more items enclosed in parentheses)

Booleans are tradition ally encoded using the symbol t for true, and the emp ty list () or the sy mbol nil for false, but for simplicity, our implementation ignores them. It als o ig nores ch annel s and functions, since their state is opaque to reflec tion. And it ignores real and complex floating-p oint numbers and int erfaces. Adding sup por t for them is Exercis e 12.3.

We’ll encode the typ es of Go using S-expressions as fol lows. Int egers and str ings are encoded in the obv iou s way. Nil values are encoded as the symbol nil. Arrays and slices are encoded using list not ation.

St ruc ts are encoded as a list of field bindings, each field binding being a two-element list whos e firs t element (a symbol) is the field name and whose second element is the field value.

Maps too are encoded as a list of pairs, wit h each pair being the key and value of one map entr y. Tradition ally, S-expressions represent lists of key/value pairs using a single cons cell (key . value) for each pair, rat her than a two-element list, but to simplif y the decoding we’ll ig nore dot ted list not ation.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 12.4. EXAMPLE: ENCODING S-EXPRESSIONS

339

Enco ding is don e by a single rec ursive function, encode, shown below. Its str ucture is ess ential ly the same as that of Display in the pre vious sec tion: gopl.io/ch12/sexpr

func encode(buf *bytes.Buffer, v reflect.Value) error {

switch v.Kind() {

case reflect.Invalid:

buf.WriteString("nil")

case reflect.Int, reflect.Int8, reflect.Int16,

reflect.Int32, reflect.Int64:

fmt.Fprintf(buf, "%d", v.Int())

case reflect.Uint, reflect.Uint8, reflect.Uint16,

reflect.Uint32, reflect.Uint64, reflect.Uintptr:

fmt.Fprintf(buf, "%d", v.Uint())

case reflect.String:

fmt.Fprintf(buf, "%q", v.String())

case reflect.Ptr:

return encode(buf, v.Elem())

ptg16091132

case reflect.Array, reflect.Slice: // (value ...)

buf.WriteByte('(')

for i := 0; i < v.Len(); i++ {

if i > 0 {

buf.WriteByte(' ')

}

if err := encode(buf, v.Index(i)); err != nil {

return err

}

}

buf.WriteByte(')')

case reflect.Struct: // ((name value) ...)

buf.WriteByte('(')

for i := 0; i < v.NumField(); i++ {

if i > 0 {

buf.WriteByte(' ')

}

fmt.Fprintf(buf, "(%s ", v.Type().Field(i).Name) if err := encode(buf, v.Field(i)); err != nil {

return err

}

buf.WriteByte(')')

}

buf.WriteByte(')')

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

340

CHAPTER 12. REFLECTION

case reflect.Map: // ((key value) ...)

buf.WriteByte('(')

for i, key := range v.MapKeys() {

if i > 0 {

buf.WriteByte(' ')

}

buf.WriteByte('(')

if err := encode(buf, key); err != nil {

return err

}

buf.WriteByte(' ')

if err := encode(buf, v.MapIndex(key)); err != nil {

return err

}

buf.WriteByte(')')

}

buf.WriteByte(')')

default: // float, complex, bool, chan, func, interface return fmt.Errorf("unsupported type: %s", v.Type())

}

return nil

}

ptg16091132

The Marshal func tion wraps the encoder in an API simi lar to those of the other encoding/... packages:

// Marshal encodes a Go value in S-expression form.

func Marshal(v interface{}) ([]byte, error) {

var buf bytes.Buffer

if err := encode(&buf, reflect.ValueOf(v)); err != nil {

return nil, err

}

return buf.Bytes(), nil

}

Here’s the out put of Marshal applie d to the strangelove var iable fro m Section 12.3: ((Title "Dr. Strangelove") (Subtitle "How I Learned to Stop Worrying and Lo ve the Bomb") (Year 1964) (Actor (("Grp. Capt. Lionel Mandrake" "Peter Sell ers") ("Pres. Merkin Muffley" "Peter Sellers") ("Gen. Buck Turgidson" "Geor ge C. Scott") ("Brig. Gen. Jack D. Ripper" "Sterling Hayden") ("Maj. T.J. \

"King\" Kong" "Slim Pickens") ("Dr. Strangelove" "Peter Sellers"))) (Oscars ("Best Actor (Nomin.)" "Best Adapted Screenplay (Nomin.)" "Best Director (N

omin.)" "Best Picture (Nomin.)")) (Sequel nil)) The whole out put app ears on one lon g line wit h minimal spaces, mak ing it hard to read.

Here’s the same out put manually for matte d according to S-expression convent ion s. Writing a pretty-print er for S-expressions is lef t as a (challenging) exercis e; the dow nlo ad from gopl.io includes a simple version.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 12.5. SETTING VARIABLES WITH REFLECT.V ALUE

341

((Title "Dr. Strangelove")

(Subtitle "How I Learned to Stop Worrying and Love the Bomb") (Year 1964)

(Actor (("Grp. Capt. Lionel Mandrake" "Peter Sellers") ("Pres. Merkin Muffley" "Peter Sellers") ("Gen. Buck Turgidson" "George C. Scott") ("Brig. Gen. Jack D. Ripper" "Sterling Hayden") ("Maj. T.J. \"King\" Kong" "Slim Pickens") ("Dr. Strangelove" "Peter Sellers"))) (Oscars ("Best Actor (Nomin.)"

"Best Adapted Screenplay (Nomin.)"

"Best Director (Nomin.)"

"Best Picture (Nomin.)"))

(Sequel nil))

Like the fmt.Print, json.Marshal, and Display func tions, sexpr.Marshal wi l l lo op forever if cal le d with cyc lic dat a.

In Sec tion 12.6, we’ll sketch out the imp lementation of the cor responding S-expression decoding function, but before we get there , we’ll firs t ne e d to underst and how reflec tion can be used to update program variables.

Exercis e 12.3: Implement the missing cas es of the encode func tion. Enco de boole ans as t and nil, floating-p oint numbers using Go’s not ation, and complex numbers like 1+2i as ptg16091132

#C(1.0 2.0). Int erfaces can be encoded as a pair of a typ e name and a value, for ins tance ("[]int" (1 2 3)), but beware that this not ation is ambiguous: the reflect.Type.String method may retur n the same str ing for dif ferent typ es.

Exercis e 12.4: Mo dif y encode to prett y-print the S-expression in the sty le shown above .

Exercis e 12.5: Ad apt encode to emit JSON ins tead of S-expressions. Test your encoder using the stand ard decoder, json.Unmarshal.

Exercis e 12.6: Ad apt encode so that, as an opt imizat ion, it does not encode a field whose value is the zero value of its typ e.

Exercis e 12.7: Create a streaming API for the S-expression decoder, fol low ing the sty le of json.Decoder (§4.5).

12.5. Setting Variables with reflect.Value

So far, reflec tion has only interpre ted values in our program in various ways. The point of this section, how ever, is to ch ange them.

Re call that som e Go expressions like x, x.f[1], and *p denot e var iables, but others like x + 1

and f(2) do not. A var iable is an ad dre ssabl e storage locat ion that contains a value, and its value may be updated through that address.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

342

CHAPTER 12. REFLECTION

A simi lar dist inc tion app lies to reflect.Values. Som e are addressable; others are not.

Consider the fol low ing declarat ions:

x := 2

// value

type

variable?

a := reflect.ValueOf(2)

// 2

int

no

b := reflect.ValueOf(x)

// 2

int

no

c := reflect.ValueOf(&x)

// &x

*int

no

d := c.Elem()

// 2

int

yes (x)

The value wit hin a is not addressable. It is merely a copy of the int eger 2. The same is true of b. The value wit hin c is als o non-addressable, being a copy of the point er value &x. In fac t, no reflect.Value returned by reflect.ValueOf(x) is addressable. But d, der ive d from c by dereferencing the point er wit hin it, refers to a var iable and is thu s addressable. We can use this appro ach, cal ling reflect.ValueOf(&x).Elem(), to obt ain an addressable Value for any var iable x.

We can ask a reflect.Value whet her it is addressable through its CanAddr method: fmt.Println(a.CanAddr()) // "false"

fmt.Println(b.CanAddr()) // "false"

fmt.Println(c.CanAddr()) // "false"

fmt.Println(d.CanAddr()) // "true"

We obt ain an addressable reflect.Value whenever we indirec t thro ugh a point er, even if we st arted fro m a non-addressable Value. All the usu al rules for addressabi lit y have analogs for ptg16091132

reflec tion. For example, since the slice indexing expression e[i] implicitly fol lows a point er, it is addressable even if the expression e is not. By analog y, reflect.ValueOf(e).Index(i) refers to a var iable, and is thu s addressable even if reflect.ValueOf(e) is not.

To recov er the var iable fro m an addressable reflect.Value re quires three steps. First, we cal l Addr(), which retur ns a Value holding a pointer to the var iable. Next, we cal l Interface() on this Value, which retur ns an interface{} value cont aining the point er. Final ly, if we know the typ e of the var iable, we can use a typ e assertion to ret rie ve the contents of the int erface as an ordinar y pointer. We can then update the var iable through the point er: x := 2

d := reflect.ValueOf(&x).Elem()

// d refers to the variable x

px := d.Addr().Interface().(*int)

// px := &x

*px = 3

// x = 3

fmt.Println(x)

// "3"

Or, we can update the var iable refer red to by an addressable reflect.Value direc tly, wit hout using a point er, by cal ling the reflect.Value.Set method: d.Set(reflect.ValueOf(4))

fmt.Println(x) // "4"

The same che cks for assig nabilit y that are ordinar ily per for med by the compi ler are don e at run time by the Set methods. Above , the var iable and the value bot h have typ e int, but if the var iable had been an int64, the program wou ld panic, so it’s cruci al to make sure the value is assig nable to the typ e of the var iable:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 12.5. SETTING VARIABLES WITH REFLECT.V ALUE

343

d.Set(reflect.ValueOf(int64(5))) // panic: int64 is not assignable to int And of cours e calling Set on a non-addressable reflect.Value panics too: x := 2

b := reflect.ValueOf(x)

b.Set(reflect.ValueOf(3)) // panic: Set using unaddressable value There are var iants of Set sp eci alized for cer tain gro ups of basic typ es: SetInt, SetUint, SetString, SetFloat, and so on:

d := reflect.ValueOf(&x).Elem()

d.SetInt(3)

fmt.Println(x) // "3"

In som e ways these met hods are more for giv ing . SetInt, for example, will succe e d so lon g as the var iable’s typ e is som e kind of sig ned int eger, or even a named typ e whos e underly ing typ e is a sig ned int eger, and if the value is too large it will be quiet ly trunc ate d to fit. But tre ad carefully: cal ling SetInt on a reflect.Value that refers to an interface{} var iable will panic, even thoug h Set would succe e d.

x := 1

rx := reflect.ValueOf(&x).Elem()

rx.SetInt(2)

// OK, x = 2

rx.Set(reflect.ValueOf(3))

// OK, x = 3

ptg16091132

rx.SetString("hello")

// panic: string is not assignable to int

rx.Set(reflect.ValueOf("hello")) // panic: string is not assignable to int var y interface{}

ry := reflect.ValueOf(&y).Elem()

ry.SetInt(2)

// panic: SetInt called on interface Value

ry.Set(reflect.ValueOf(3))

// OK, y = int(3)

ry.SetString("hello")

// panic: SetString called on interface Value

ry.Set(reflect.ValueOf("hello")) // OK, y = "hello"

Wh en we app lie d Display to os.Stdout, we found that reflec tion can read the values of unexp orted str uct fields that are inaccessible according to the usu al rules of the langu age, like the fd int field of an os.File st ruc t on a Unix-li ke platfor m. How ever, reflec tion cannot up date such values:

stdout := reflect.ValueOf(os.Stdout).Elem() // *os.Stdout, an os.File var fmt.Println(stdout.Type())

// "os.File"

fd := stdout.FieldByName("fd")

fmt.Println(fd.Int()) // "1"

fd.SetInt(2)

// panic: unexported field

An addressable reflect.Value re cords whether it was obt ained by traversing an unexp orted st ruc t field and, if so, dis allows modification. Con sequently, CanAddr is not usu ally the rig ht ch eck to use before setting a var iable. The rel ate d method CanSet reports whether a reflect.Value is addressable an d sett able:

fmt.Println(fd.CanAddr(), fd.CanSet()) // "true false"

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

344

CHAPTER 12. REFLECTION

12.6. Example: Decoding S-Expressions

Fo r each Marshal func tion provide d by the stand ard librar y’s encoding/... packages, there is a cor responding Unmarshal func tion that does decoding. For example, as we saw in Section 4.5, given a byte slice cont aining JSON-enco ded dat a for our Movie type (§12.3), we can decode it like this:

data := []byte{/* ... */}

var movie Movie

err := json.Unmarshal(data, &movie)

The Unmarshal func tion uses reflec tion to modif y the fields of the exist ing movie var iable, creating new maps, str ucts, and slices as deter mined by the typ e Movie and the content of the incoming dat a.

Let’s now imp lement a simple Unmarshal func tion for S-expressions, analogou s to the standard json.Unmarshal func tion used above , and the inv ers e of our earlier sexpr.Marshal.

We must caution you that a robust and general implementation requires subst ant ial ly more co de than will comfortably fit in this example, which is already lon g, so we have taken many shortcuts. We sup por t on ly a limite d su bset of S-expressions and do not handle error s grace-fully. The code is int ended to illustrate reflec tion, not parsing.

The lexer uses the Scanner type fro m the text/scanner package to bre ak an inp ut stream ptg16091132

into a sequence of tokens such as comments, identifiers, str ing lit erals, and numer ic literals.

The scanner’s Scan method advances the scanner and retur ns the kind of the next token, which has typ e rune. Most tokens, like '(', con sist of a single rune, but the text/scanner package represents the kinds of the multi-charac ter tokens Ident, String, and Int using smal l negat ive values of typ e rune. Fol low ing a cal l to Scan that retur ns one of these kinds of token, the scanner’s TokenText method retur ns the text of the token.

Since a typical parser may need to ins pec t the cur rent token several times, but the Scan method advances the scanner, we wrap the scanner in a helper typ e called lexer that keeps track of the token most recently retur ned by Scan.

gopl.io/ch12/sexpr

type lexer struct {

scan scanner.Scanner

token rune // the current token

}

func (lex *lexer) next()

{ lex.token = lex.scan.Scan() }

func (lex *lexer) text() string { return lex.scan.TokenText() }

func (lex *lexer) consume(want rune) {

if lex.token != want { // NOTE: Not an example of good error handling.

panic(fmt.Sprintf("got %q, want %q", lex.text(), want))

}

lex.next()

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 12.6. EXAMPLE: DECODING S-EXPRESSIONS

345

No w let’s tur n to the parser. It con sists of two princip al functions. The firs t of these, read, re ads the S-expression that starts wit h the cur rent token and updates the var iable refer red to by the addressable reflect.Value v.

func read(lex *lexer, v reflect.Value) {

switch lex.token {

case scanner.Ident:

// The only valid identifiers are

// "nil" and struct field names.

if lex.text() == "nil" {

v.Set(reflect.Zero(v.Type()))

lex.next()

return

}

case scanner.String:

s, _ := strconv.Unquote(lex.text()) // NOTE: ignoring errors v.SetString(s)

lex.next()

return

case scanner.Int:

i, _ := strconv.Atoi(lex.text()) // NOTE: ignoring errors v.SetInt(int64(i))

lex.next()

return

ptg16091132

case '(':

lex.next()

readList(lex, v)

lex.next() // consume ')'

return

}

panic(fmt.Sprintf("unexpected token %q", lex.text()))

}

Our S-expressions use identifiers for two distinc t purpos es, st ruc t field names and the nil value for a point er. The read func tion only handles the latter cas e. When it encount ers the scanner.Ident "nil", it sets v to the zero value of its typ e using the reflect.Zero func tion.

Fo r any other identifier, it rep orts an error. The readList func tion, which we’ll see in a moment, handles identifiers used as str uct field names.

A '(' token indic ates the start of a list. The secon d func tion, readList, decodes a list int o a var iable of composite typ e—a map, str uct, slice, or array—dep ending on what kind of Go var iable we’re cur rently popu lat ing. In each cas e, the loop keeps parsing items unt i l it encounters the matching clos e parent hesis, ')', as detec ted by the endList func tion.

The int erest ing par t is the rec ursion. The simplest cas e is an array. Unt i l the closing ')' is seen, we use Index to obt ain the var iable for each array element and make a rec ursive cal l to read to popu late it. As in many other erro r cases, if the inp ut dat a caus es the decoder to index beyond the end of the array, the decoder panics. A simi lar approach is used for slices, except we must cre ate a new var iable for each element, popu late it, then app end it to the slice.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

346

CHAPTER 12. REFLECTION

The loops for str ucts and maps must parse a (key value) su blist on each iterat ion. For st ruc ts, the key is a symbol identifying the field. Analogou s to the cas e for arrays, we obt ain the exist ing var iable for the str uct field using FieldByName and make a rec ursive cal l to popu late it. For maps, the key may be of any typ e, and analogou s to the cas e for slices, we create a new var iable, rec ursively popu late it, and finally ins ert the new key/value pair into the map.

func readList(lex *lexer, v reflect.Value) {

switch v.Kind() {

case reflect.Array: // (item ...)

for i := 0; !endList(lex); i++ {

read(lex, v.Index(i))

}

case reflect.Slice: // (item ...)

for !endList(lex) {

item := reflect.New(v.Type().Elem()).Elem()

read(lex, item)

v.Set(reflect.Append(v, item))

}

case reflect.Struct: // ((name value) ...)

for !endList(lex) {

ptg16091132

lex.consume('(')

if lex.token != scanner.Ident {

panic(fmt.Sprintf("got token %q, want field name", lex.text()))

}

name := lex.text()

lex.next()

read(lex, v.FieldByName(name))

lex.consume(')')

}

case reflect.Map: // ((key value) ...)

v.Set(reflect.MakeMap(v.Type()))

for !endList(lex) {

lex.consume('(')

key := reflect.New(v.Type().Key()).Elem()

read(lex, key)

value := reflect.New(v.Type().Elem()).Elem()

read(lex, value)

v.SetMapIndex(key, value)

lex.consume(')')

}

default:

panic(fmt.Sprintf("cannot decode list into %v", v.Type()))

}

}

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 12.6. EXAMPLE: DECODING S-EXPRESSIONS

347

func endList(lex *lexer) bool {

switch lex.token {

case scanner.EOF:

panic("end of file")

case ')':

return true

}

return false

}

Final ly, we wrap up the parser in an exp orted function Unmarshal, shown below, that hides some of the rough edges of the imp lementation. Error s encountere d during parsing result in a panic, so Unmarshal us es a defer red cal l to recov er fro m the panic (§5.10) and retur n an erro r mess age ins tead.

// Unmarshal parses S-expression data and populates the variable

// whose address is in the non-nil pointer out.

func Unmarshal(data []byte, out interface{}) (err error) {

lex := &lexer{scan: scanner.Scanner{Mode: scanner.GoTokens}}

lex.scan.Init(bytes.NewReader(data))

lex.next() // get the first token

defer func() {

// NOTE: this is not an example of ideal error handling.

if x := recover(); x != nil {

ptg16091132

err = fmt.Errorf("error at %s: %v", lex.scan.Position, x)

}

}()

read(lex, reflect.ValueOf(out).Elem())

return nil

}

A pro duc tion-quality imp lementation should never panic for any inp ut and should rep ort an infor mat ive error for every mishap, perhaps wit h a line number or offset. Non etheless, we hope this example conve ys som e ide a of what’s happening under the hood of the packages like encoding/json, and how you can use reflec tion to popu late dat a st ruc tures.

Exercis e 12.8: The sexpr.Unmarshal func tion, like json.Marshal, requires the complete input in a byte slice before it can beg in de coding. Define a sexpr.Decoder type that, like json.Decoder, allows a sequence of values to be decoded fro m an io.Reader. Change sexpr.Unmarshal to use this new typ e.

Exercis e 12.9: Wr ite a token-b ased API for decoding S-expressions, fol low ing the sty le of xml.Decoder (§7.14). You will need five typ es of tokens: Symbol, String, Int, StartList, and EndList.

Exercis e 12.10: Extend sexpr.Unmarshal to handle the boole ans, floating-p oint numbers, and int erfaces enco ded by your solut ion to Exercis e 12.3. (Hint: to decode int erfaces, you will ne e d a mapping fro m the name of each sup por ted typ e to its reflect.Type.)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

348

CHAPTER 12. REFLECTION

12.7. Accessing Struct Field Tags

In Sec tion 4.5 we used str uct fiel d tags to modif y the JSON encoding of Go str uct values. The json field tag lets us cho ose alt ernat ive field names and sup press the out put of emp ty fields. In this sec tion, we’ll see how to access field tags using reflec tion.

In a web ser ver, the firs t thing most HTTP handler functions do is ext ract the request parameters int o lo cal variables. We’ll define a utilit y func tion, params.Unpack, that uses str uct field tags to make writing HTTP handlers (§7.7) more convenient.

First, we’ll show how it’s used. The search func tion below is an HTTP handler. It defines a var iable cal le d data of an anony mou s st ruc t type whose fields cor respond to the HTTP

re quest parameters. The str uct’s field tags specif y the parameter names, which are often short and crypt ic since space is pre cious in a URL. The Unpack func tion popu lates the str uct fro m the request so that the parameters can be accessed conveniently and wit h an appropriate typ e.

gopl.io/ch12/search

import "gopl.io/ch12/params"

// search implements the /search URL endpoint.

func search(resp http.ResponseWriter, req *http.Request) {

var data struct {

ptg16091132

Labels

[]string `http:"l"`

MaxResults int

`http:"max"Èxact

bool

`http:"x"`

}

data.MaxResults = 10 // set default

if err := params.Unpack(req, &data); err != nil {

http.Error(resp, err.Error(), http.StatusBadRequest) // 400

return

}

// ...rest of handler...

fmt.Fprintf(resp, "Search: %+v\n", data)

}

The Unpack func tion below does three things. First, it cal ls req.ParseForm() to parse the re quest. Thereafter, req.Form cont ains all the parameters, regardless of whether the HTTP

client used the GET or the POST request met hod.

Next, Unpack builds a mapping fro m the effec tiv e name of each field to the var iable for that field. The effec tive name may differ fro m the actual name if the field has a tag . The Field method of reflect.Type returns a reflect.StructField that provides infor mat ion about the typ e of each field such as its name, typ e, and opt ion al tag . The Tag field is a reflect.StructTag, which is a str ing typ e that provides a Get method to parse and ext ract the subst ring for a par tic ular key, such as http:"..." in this cas e.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 12.7. ACCESSING STRUCT FIELD TAGS

349

gopl.io/ch12/params

// Unpack populates the fields of the struct pointed to by ptr

// from the HTTP request parameters in req.

func Unpack(req *http.Request, ptr interface{}) error {

if err := req.ParseForm(); err != nil {

return err

}

// Build map of fields keyed by effective name.

fields := make(map[string]reflect.Value)

v := reflect.ValueOf(ptr).Elem() // the struct variable for i := 0; i < v.NumField(); i++ {

fieldInfo := v.Type().Field(i) // a reflect.StructField tag := fieldInfo.Tag

// a reflect.StructTag

name := tag.Get("http")

if name == "" {

name = strings.ToLower(fieldInfo.Name)

}

fields[name] = v.Field(i)

}

// Update struct field for each parameter in the request.

for name, values := range req.Form {

f := fields[name]

ptg16091132

if !f.IsValid() {

continue // ignore unrecognized HTTP parameters

}

for _, value := range values {

if f.Kind() == reflect.Slice {

elem := reflect.New(f.Type().Elem()).Elem()

if err := populate(elem, value); err != nil {

return fmt.Errorf("%s: %v", name, err)

}

f.Set(reflect.Append(f, elem))

} else {

if err := populate(f, value); err != nil {

return fmt.Errorf("%s: %v", name, err)

}

}

}

}

return nil

}

Final ly, Unpack it erates over the name/value pairs of the HTTP parameters and updates the corresponding str uct fields. Recal l that the same parameter name may appear more than on ce. If this happens, and the field is a slice, then all the values of that parameter are acc umulate d into the slice. Other wis e, the field is rep eatedly overwritt en so that only the last value has any effec t.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

350

CHAPTER 12. REFLECTION

The populate func tion takes care of setting a single field v (or a single element of a slice field) from a parameter value. For now, it sup por ts only str ings, signe d integers, and boole ans.

Su pporting other typ es is lef t as an exercise.

func populate(v reflect.Value, value string) error {

switch v.Kind() {

case reflect.String:

v.SetString(value)

case reflect.Int:

i, err := strconv.ParseInt(value, 10, 64)

if err != nil {

return err

}

v.SetInt(i)

case reflect.Bool:

b, err := strconv.ParseBool(value)

if err != nil {

return err

}

v.SetBool(b)

default:

return fmt.Errorf("unsupported kind %s", v.Type()) ptg16091132

}

return nil

}

If we add the server hand ler to a web ser ver, this mig ht be a typic al session: $ go build gopl.io/ch12/search

$ ./search &

$ ./fetch 'http://localhost:12345/search'

Search: {Labels:[] MaxResults:10 Exact:false}

$ ./fetch 'http://localhost:12345/search?l=golang&l=programming'

Search: {Labels:[golang programming] MaxResults:10 Exact:false}

$ ./fetch 'http://localhost:12345/search?l=golang&l=programming&max=100'

Search: {Labels:[golang programming] MaxResults:100 Exact:false}

$ ./fetch 'http://localhost:12345/search?x=true&l=golang&l=programming'

Search: {Labels:[golang programming] MaxResults:10 Exact:true}

$ ./fetch 'http://localhost:12345/search?q=hello&x=123'

x: strconv.ParseBool: parsing "123": invalid syntax $ ./fetch 'http://localhost:12345/search?q=hello&max=lots'

max: strconv.ParseInt: parsing "lots": invalid syntax Exercis e 12.11: Wr ite the cor responding Pack func tion. Given a str uct value, Pack should return a URL incorporat ing the parameter values fro m the str uct.

Exercis e 12.12: Extend the field tag not ation to express parameter validity requirements. For example, a str ing mig ht need to be a valid emai l address or cre dit-c ard number, and an int eger mig ht need to be a valid US ZIP code. Modif y Unpack to check these requirements.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 12.8. DISPLAYING THE METHODS OF A TYPE

351

Exercis e 12.13: Mo dif y the S-expression encoder (§12.4) and decoder (§12.6) so that the y honor the sexpr:"..." field tag in a similar manner to encoding/json (§4.5).

12.8. Displaying the Methods of a Type

Our final example of reflec tion uses reflect.Type to print the typ e of an arbit rar y value and enumerate its met hods:

gopl.io/ch12/methods

// Print prints the method set of the value x.

func Print(x interface{}) {

v := reflect.ValueOf(x)

t := v.Type()

fmt.Printf("type %s\n", t)

for i := 0; i < v.NumMethod(); i++ {

methType := v.Method(i).Type()

fmt.Printf("func (%s) %s%s\n", t, t.Method(i).Name, strings.TrimPrefix(methType.String(), "func"))

}

ptg16091132

}

Both reflect.Type and reflect.Value have a met hod cal le d Method. Each t.Method(i) call retur ns an ins tance of reflect.Method, a str uct typ e that descr ibes the name and typ e of a single met hod. Each v.Method(i) call retur ns a reflect.Value repres enting a met hod value (§6.4), that is, a met hod bound to its receiver. Using the reflect.Value.Call method (w hich we don’t have space to show here), it’s possible to cal l Values of kind Func li ke this one, but this program needs only its Type.

Here are the methods belon ging to two typ es, time.Duration and *strings.Replacer: methods.Print(time.Hour)

// Output:

// type time.Duration

// func (time.Duration) Hours() float64

// func (time.Duration) Minutes() float64

// func (time.Duration) Nanoseconds() int64

// func (time.Duration) Seconds() float64

// func (time.Duration) String() string

methods.Print(new(strings.Replacer))

// Output:

// type *strings.Replacer

// func (*strings.Replacer) Replace(string) string

// func (*strings.Replacer) WriteString(io.Writer, string) (int, error)

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

352

CHAPTER 12. REFLECTION

12.9. A Word of Caution

There is a lot more to the reflec tion API than we have space to show, but the pre ceding examples give an ide a of what is possible. Reflec tion is a pow erful and expressive tool, but it should be used wit h care , for three reasons.

The firs t re ason is that reflec tion-b ased code can be frag i le. For every mistake that wou ld caus e a compi ler to rep ort a typ e er ror, there is a cor responding way to misus e reflec tion, but whereas the compi ler rep orts the mistake at bui ld time, a reflec tion error is rep orted dur ing exec ution as a panic, possibly lon g af ter the program was writt en or even lon g af ter it has st arted running.

If the readList func tion (§12.6), for example, should read a str ing fro m the inp ut whi le popu lat ing a var iable of typ e int, the cal l to reflect.Value.SetString wi l l panic. Most prog rams that use reflec tion have simi lar hazards, and con siderable care is required to keep track of the typ e, addressabi lit y, and settabi lit y of each reflect.Value.

The best way to avoid this frag i lity is to ens ure that the use of reflec tion is fully enc apsulated within your package and, if possible, avoid reflect.Value in favor of specific typ es in your package’s API, to restr ict inp uts to legal values. If this is not possible, per for m addition al dy namic che cks before each risky operat ion. As an example fro m the stand ard librar y, when fmt.Printf applies a verb to an inappro priate operand, it does not panic mysteriou sly but ptg16091132

pr ints an infor mat ive error message . The program still has a bug , but it is easier to diagnos e.

fmt.Printf("%d %s\n", "hello", 42) // "%!d(string=hello) %!s(int=42)"

Reflec tion als o re duces the safet y and acc uracy of aut omated refac tor ing and analysis tools, becaus e they can’t deter mine or rely on typ e infor mat ion.

The secon d re ason to avoid reflec tion is that since typ es serve as a for m of documentation and the operat ions of reflec tion cannot be subjec t to static typ e ch eck ing , he avi ly reflec tive code is of ten hard to underst and. Always caref ully document the exp ected typ es and other invar iants of functions that accept an interface{} or a reflect.Value.

The third reason is that reflec tion-b ased functions may be one or two orders of mag nitude slow er than code speci alized for a par tic ular typ e. In a typic al prog ram, the maj ority of function s are not relevant to the overal l perfor mance, so it’s fine to use reflec tion when it makes the prog ram clearer. Testing is a par tic ularly good fit for reflec tion since most tests use small dat a sets. But for functions on the critical pat h, reflec tion is best avoide d.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

13

Lo w-Level Pro gramming

The desig n of Go guarante es a number of safet y prop erties that limit the ways in which a Go prog ram can ‘‘go wro ng.’’ During compi lat ion, typ e ch eck ing detec ts most att emp ts to app l y an operat ion to a value that is inappro priate for its typ e, for ins tance, subtrac ting one str ing from another. Str ict rules for typ e conv ersions pre vent direc t access to the int ernals of bui lt-in ptg16091132

types like str ings, maps, slices, and channel s.

Fo r er ror s that cannot be detec ted statically, such as out-of-b ounds array accesses or nil pointer dereferences, dynamic che cks ensure that the program immediate ly ter minates wit h an infor mat ive error whene ver a forbidden operat ion occ urs. Aut omat ic memory management (garb age col lec tion) eliminates ‘‘us e af ter fre e’’ bugs, as wel l as most memor y le aks.

Many imp lementation det ails are inaccessible to Go programs. There is no way to dis cov er the memor y layout of an aggregate typ e li ke a str uct, or the machine code for a function, or the identity of the operat ing system thread on which the cur rent goroutine is running.

In deed, the Go sch edu ler fre ely mov es goroutines fro m on e thre ad to another. A pointer identifies a variable wit hout revealing the var iable’s numer ic address. Addresses may change as the garb age col lec tor mov es var iables; point ers are transp arent ly updated.

Together, these features make Go programs, especi ally fai ling ones, more pre dic table and less myster ious than programs in C, the quintess ent ial low-le vel langu age. By hiding the underly ing det ails, the y also make Go programs hig hly por table, since the langu age semantics are large ly indep endent of any par tic ular compi ler, operat ing system, or CPU archit ecture. (No t entire ly indep endent: som e det ails leak through, such as the word size of the pro cessor, the order of evaluation of cer tain expressions, and the set of imp lementation restr ictions imp osed by the compi ler.)

Occasionally, we may cho ose to for feit som e of these helpf ul guarante es to achie ve the hig hest possible per for mance, to int ero perate wit h librar ies wr itt en in other langu ages, or to imp lement a function that cannot be express ed in pure Go.

353

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

354

CHAPTER 13. LOW-LEVEL PROGRAMMING

In this chapt er, we’ll see how the unsafe package lets us step outside the usu al rules, and how to use the cgo to ol to cre ate Go bindings for C librar ies and operat ing system cal ls.

The appro ach es des crib ed in this chapt er should not be used frivo lou sly. Wit hout caref ul attent ion to det ail, the y may cause the kinds of unp redic table, ins crutable, non-lo cal failures with which C programmers are unhappi ly acquainte d. Us e of unsafe also voids Go’s war ranty of compatibi lit y with future rele ases, since, whether intended or inadver tent, it is easy to dep end on uns pecified imp lementation det ails that may change unexp ectedly.

The unsafe package is rat her mag ical. Alt hough it app ears to be a regu lar package and is imported in the usu al way, it is actually imp lemente d by the compi ler. It provides access to a numb er of bui lt-in langu age features that are not ordinar ily avai lable because the y exp ose det ails of Go’s memor y layout. Present ing these features as a sep arate package makes the rare occasions on which the y are needed more con spicuou s. Als o, som e enviro nments may restr ict the use of the unsafe package for sec urity reasons.

Package unsafe is used extensive ly wit hin low-le vel packages like runtime, os, syscall, and net that interac t with the operat ing system, but is almost never needed by ordinar y prog rams.

13.1. unsafe.Sizeof, Alignof, and Offsetof

ptg16091132

The unsafe.Sizeof func tion rep orts the size in bytes of the represent ation of its operand, which may be an expression of any typ e; the expression is not evaluate d. A cal l to Sizeof is a cons tant expression of typ e uintptr, so the result may be used as the dimension of an array type, or to compute other cons tants.

import "unsafe"

fmt.Println(unsafe.Sizeof(float64(0))) // "8"

Sizeof reports only the size of the fixe d part of each dat a st ruc ture, like the point er and lengt h of a str ing , but not indirec t parts like the contents of the str ing . Typical sizes for all non-ag gregate Go typ es are shown below, thoug h the exac t sizes may vary by toolchain. For portabilit y, we’ve given the sizes of reference typ es (or typ es cont aining references) in ter ms of word s, where a word is 4 bytes on a 32-bit platfor m and 8 bytes on a 64-bit platfor m.

Comp uters load and store values fro m memory most efficiently when those values are prop erly alig ned. For example, the address of a value of a two-byte typ e such as int16 should be an even number, the address of a four-byte value such as a rune should be a multiple of four, and the address of an eig ht-byte value such as a float64, uint64, or 64-bit point er should be a multiple of eig ht. Alig nment requirements of hig her multiples are unu sual, even for larger dat a types such as complex128.

Fo r this reason, the size of a value of an aggregate typ e (a str uct or array) is at least the sum of the sizes of its fields or elements but may be gre ater due to the presence of ‘‘holes.’’ Ho les are unus ed spaces added by the compi ler to ens ure that the fol low ing field or element is pro perly alig ned rel ative to the start of the str uct or array.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 13.1. UNSAFE.SIZEOF, ALIGNOF, AND OFFSETOF

355

Ty pe

Siz e

bool

1 byte

intN, uintN, floatN, complexN

N / 8 bytes (for example, float64 is 8 bytes)

int, uint, uintptr

1 word

*T

1 word

string

2 words (data, len)

[]T

3 words (data, len, cap)

map

1 word

func

1 word

chan

1 word

interface

2 words (type, value)

The langu age specification does not guarante e that the order in which fields are declare d is the order in which the y are laid out in memor y, so in the ory a compi ler is fre e to rearrange them, although as we write this, non e do. If the typ es of a str uct’s fields are of dif ferent sizes, it may be more space-efficient to declare the fields in an order that packs them as tig htly as possible.

The three str ucts below have the same fields, but the firs t re quires up to 50% more memor y than the other two:

// 64-bit

32-bit

struct{ bool; float64; int16 }

// 3 words

4 words

struct{ float64; int16; bool }

// 2 words

3 words

ptg16091132

struct{ bool; int16; float64 }

// 2 words

3 words

The det ails of the alig nment algor it hm are beyon d the scope of this book, and it’s cer tain ly not worth wor rying about every str uct, but efficient packing may make fre quently allocated dat a st ruc tures more compact and therefore faster.

The unsafe.Alignof func tion rep orts the required alig nment of its argument’s typ e. Like Sizeof, it may be app lie d to an expression of any typ e, and it yields a con stant. Typic ally, boole an and numer ic types are alig ned to their size (up to a maximum of 8 bytes) and all other types are word-alig ned.

The unsafe.Offsetof func tion, whose operand must be a field selec tor x.f, computes the of fset of field f re lat ive to the start of its enclosing str uct x, accounting for holes, if any.

Figure 13.1 shows a str uct var iable x and its memor y layout on typic al 32- and 64-bit Go implementation s. The gray reg ions are holes.

var x struct {

a bool

b int16

c []int

}

The table below shows the results of app l ying the three unsafe func tions to x itself and to each of its three fields:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

356

CHAPTER 13. LOW-LEVEL PROGRAMMING

Figure 13.1. Ho les in a str uct.

Typical 32-bit platform:

Sizeof(x)

= 16 Alignof(x)

= 4

Sizeof(x.a) = 1

Alignof(x.a) = 1

Offsetof(x.a) = 0

Sizeof(x.b) = 2

Alignof(x.b) = 2

Offsetof(x.b) = 2

Sizeof(x.c) = 12

Alignof(x.c) = 4

Offsetof(x.c) = 4

Typical 64-bit platform:

Sizeof(x)

= 32 Alignof(x)

= 8

Sizeof(x.a) = 1

Alignof(x.a) = 1

Offsetof(x.a) = 0

Sizeof(x.b) = 2

Alignof(x.b) = 2

Offsetof(x.b) = 2

ptg16091132

Sizeof(x.c) = 24

Alignof(x.c) = 8

Offsetof(x.c) = 8

Despit e their names, these functions are not in fac t unsafe , and the y may be helpf ul for underst anding the layout of raw memor y in a program when opt imizing for space.

13.2. unsafe.Pointer

Most point er typ es are writt en *T, meaning ‘‘a point er to a var iable of typ e T.’’ The unsafe.Pointer type is a speci al kind of point er that can hold the address of any var iable. Of course, we can’t indirec t thro ugh an unsafe.Pointer using *p becaus e we don’t know what type that expression should have. Like ordinar y pointers, unsafe.Pointers are comparable and may be compare d with nil, which is the zero value of the typ e.

An ordinar y *T pointer may be converted to an unsafe.Pointer, and an unsafe.Pointer may be converted back to an ordinar y pointer, not necessarily of the same typ e *T. By converting a *float64 pointer to a *uint64, for ins tance, we can insp ect the bit pattern of a floating-p oint var iable:

package math

func Float64bits(f float64) uint64 { return *(*uint64)(unsafe.Pointer(&f)) }

fmt.Printf("%#016x\n", Float64bits(1.0)) // "0x3ff0000000000000"

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 13.2. UNSAFE.POINTER

357

Through the resulting point er, we can update the bit pattern too. This is har mless for a floating-p oint var iable since any bit pattern is legal, but in general, unsafe.Pointer conv ersions let us write arbit rar y values to memor y and thu s su bvert the typ e system.

An unsafe.Pointer may als o be converted to a uintptr that holds the point er’s numer ic value, letting us per for m ar ithmetic on addresses. (Recal l from Chapt er 3 that a uintptr is an unsig ned int eger wide enough to represent an address.) This conversion too may be app lie d in reverse, but again, conv erting fro m a uintptr to an unsafe.Pointer may subvert the typ e system since not all numbers are valid addresses.

Many unsafe.Pointer values are thu s intermediaries for converting ordinar y pointers to raw numeric addresses and back again. The example below takes the address of var iable x, adds the offset of its b field, converts the resulting address to *int16, and through that point er up dates x.b:

gopl.io/ch13/unsafeptr

var x struct {

a bool

b int16

c []int

}

// equivalent to pb := &x.b

ptg16091132

pb := (*int16)(unsafe.Pointer(

uintptr(unsafe.Pointer(&x)) + unsafe.Offsetof(x.b)))

*pb = 42

fmt.Println(x.b) // "42"

Although the syntax is cumbers ome—perh aps no bad thing since these features should be us ed sparing ly—do not be tempt ed to int roduce temporar y var iables of typ e uintptr to bre ak the lines. This code is incor rec t:

// NOTE: subtly incorrect!

tmp := uintptr(unsafe.Pointer(&x)) + unsafe.Offsetof(x.b) pb := (*int16)(unsafe.Pointer(tmp))

*pb = 42

The reason is ver y su btle. Som e garb age col lec tor s move var iables around in memor y to re duce frag mentation or bookkeeping . Garb age col lec tor s of this kind are known as moving GCs. When a var iable is mov ed, all point ers that hold the address of the old location must be up dated to point to the new one. Fro m the persp ective of the garb age col lec tor, an unsafe.Pointer is a point er and thu s its value must change as the var iable mov es, but a uintptr is just a number so its value must not change . The incor rec t co de ab ove hi d es a pointer from the garb age col lec tor in the non-p ointer var iable tmp. By the time the secon d st atement exe cut es, the var iable x could have mov ed and the number in tmp would no lon g er be the address &x.b. The third statement clobb ers an arbit rar y memory locat ion wit h the value 42.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

358

CHAPTER 13. LOW-LEVEL PROGRAMMING

There are myr iad pat holog ical variation s on this theme. After this statement has exec ute d: pT := uintptr(unsafe.Pointer(new(T))) // NOTE: wrong!

there are no point ers that refer to the var iable create d by new, so the garb age col lec tor is entitled to rec ycle its storage when this statement completes, after which pT cont ains the address where the var iable was but is no lon g er.

No cur rent Go imp lementation uses a mov ing garb age col lec tor (though fut ure imp lementation s mig ht), but this is no reason for compl acency : current versions of Go do mov e some var iables around in memor y. Recal l from Sec tion 5.2 that goroutine stacks grow as needed.

Wh en this happens, all var iables on the old stack may be relocated to a new, larger stack, so we cannot rely on the numer ic value of a var iable’s address remaining unchange d thro ughout its lifet ime.

At the time of writing , there is lit tle cle ar guid ance on what Go programmers may rely upon af ter an unsafe.Pointer to uintptr conv ersion (see Go issue 7192), so we stron gly recommend that you assume the bare minimum. Tre at all uintptr values as if the y cont ain the for mer address of a var iable, and minimize the number of operat ions bet ween converting an unsafe.Pointer to a uintptr and using that uintptr. In our firs t example above , the three op erat ions—conv ersion to a uintptr, addition of the field offset, conv ersion back—al l appeared wit hin a single expression.

Wh en cal ling a librar y func tion that retur ns a uintptr, such as those below fro m the reflect ptg16091132

package, the result should be immediate ly converted to an unsafe.Pointer to ens ure that it cont inues to point to the same var iable.

package reflect

func (Value) Pointer() uintptr

func (Value) UnsafeAddr() uintptr

func (Value) InterfaceData() [2]uintptr // (index 1)

13.3. Example: Deep Equivalence

The DeepEqual func tion fro m the reflect package rep orts whether two values are ‘‘de eply’’

equal. DeepEqual comp ares basic values as if by the bui lt-in == op erator ; for composite values, it traverses them rec ursively, comparing cor responding elements. Because it wor ks for any pair of values, even ones that are not comparable wit h ==, it find s widespread use in tests.

The fol low ing test uses DeepEqual to compare two []string values: func TestSplit(t *testing.T) {

got := strings.Split("a:b:c", ":")

want := []string{"a", "b", "c"}; if !reflect.DeepEqual(got, want) { /* ... */ }

}

Although DeepEqual is convenient, its dist inc tions can seem arbit rar y. For example, it doesn’t consider a nil map equ al to a non-ni l empt y map, nor a nil slice equ al to a non-ni l empt y on e:

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 13.3. EXAMPLE: DEEP EQUIVALENCE

359

var a, b []string = nil, []string{}

fmt.Println(reflect.DeepEqual(a, b)) // "false"

var c, d map[string]int = nil, make(map[string]int)

fmt.Println(reflect.DeepEqual(c, d)) // "false"

In this sec tion we’ll define a function Equal that comp ares arbit rar y values. Like DeepEqual, it compares slices and maps bas ed on their elements, but unlike DeepEqual, it con siders a nil slice (or map) equ al to a non-ni l empt y on e. The basic rec ursion over the arguments can be done wit h reflec tion, using a simi lar approach to the Display prog ram we saw in Sec tion 12.3.

As usu al, we define an unexp orted function, equal, for the rec ursion. Don’t wor ry about the seen parameter just yet. For each pair of values x and y to be compare d, equal ch ecks that both (or neither) are valid and che cks that the y have the same typ e. The result of the function is define d as a set of switch cas es that comp are two values of the same typ e. For reasons of sp ace, we’ve omitt ed several cas es since the pattern should be fami liar by now.

gopl.io/ch13/equal

func equal(x, y reflect.Value, seen map[comparison]bool) bool {

if !x.IsValid() || !y.IsValid() {

return x.IsValid() == y.IsValid()

}

if x.Type() != y.Type() {

return false

}

ptg16091132

// ...cycle check omitted (shown later)...

switch x.Kind() {

case reflect.Bool:

return x.Bool() == y.Bool()

case reflect.String:

return x.String() == y.String()

// ...numeric cases omitted for brevity...

case reflect.Chan, reflect.UnsafePointer, reflect.Func: return x.Pointer() == y.Pointer()

case reflect.Ptr, reflect.Interface:

return equal(x.Elem(), y.Elem(), seen)

case reflect.Array, reflect.Slice:

if x.Len() != y.Len() {

return false

}

for i := 0; i < x.Len(); i++ {

if !equal(x.Index(i), y.Index(i), seen) {

return false

}

}

return true

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

360

CHAPTER 13. LOW-LEVEL PROGRAMMING

// ...struct and map cases omitted for brevity...

}

panic("unreachable")

}

As usu al, we don’t exp ose the use of reflec tion in the API, so the exp orted function Equal mu st cal l reflect.ValueOf on its arguments:

// Equal reports whether x and y are deeply equal.

func Equal(x, y interface{}) bool {

seen := make(map[comparison]bool)

return equal(reflect.ValueOf(x), reflect.ValueOf(y), seen)

}

type comparison struct {

x, y unsafe.Pointer

t

reflect.Type

}

To ens ure that the algor it hm ter minates even for cyc lic dat a st ruc tures, it must record which pairs of var iables it has already compare d and avoid comp aring them a secon d time. Equal al locates a set of comparison st ruc ts, each holding the address of two var iables (represent ed as unsafe.Pointer values) and the typ e of the comparison. We need to record the typ e in addition to the addresses because dif ferent var iables can have the same address. For example, if x ptg16091132

and y are bot h ar rays, x and x[0] have the same address, as do y and y[0], and it is imp ortant to distinguish whether we have compare d x and y or x[0] and y[0].

Once equal has est ablishe d that its arguments have the same typ e, and before it exe cut es the sw itch, it che cks whether it is comparing two var iables it has already seen and, if so, ter minates the rec ursion.

// cycle check

if x.CanAddr() && y.CanAddr() {

xptr := unsafe.Pointer(x.UnsafeAddr())

yptr := unsafe.Pointer(y.UnsafeAddr())

if xptr == yptr {

return true // identical references

}

c := comparison{xptr, yptr, x.Type()}

if seen[c] {

return true // already seen

}

seen[c] = true

}

Here’s our Equal func tion in action:

fmt.Println(Equal([]int{1, 2, 3}, []int{1, 2, 3}))

// "true"

fmt.Println(Equal([]string{"foo"}, []string{"bar"}))

// "false"

fmt.Println(Equal([]string(nil), []string{}))

// "true"

fmt.Println(Equal(map[string]int(nil), map[string]int{})) // "true"

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 13.4. CALLING C CODE WITH CGO

361

It even wor ks on cyc lic inputs similar to the one that caused the Display func tion fro m Section 12.3 to get stuck in a loop:

// Circular linked lists a -> b -> a and c -> c.

type link struct {

value string

tail *link

}

a, b, c := &link{value: "a"}, &link{value: "b"}, &link{value: "c"}

a.tail, b.tail, c.tail = b, a, c

fmt.Println(Equal(a, a)) // "true"

fmt.Println(Equal(b, b)) // "true"

fmt.Println(Equal(c, c)) // "true"

fmt.Println(Equal(a, b)) // "false"

fmt.Println(Equal(a, c)) // "false"

Exercis e 13.1: Define a deep comparison function that con siders numbers (of any typ e) equal if the y dif fer by less than one par t in a billion.

Exercis e 13.2: Wr ite a function that rep orts whether its argument is a cyc lic dat a st ruc ture.

13.4. Calling C Code with cgo

ptg16091132

A Go program might need to use a hardware driver imp lemente d in C, quer y an embedde d database imp lemente d in C++, or use some linear algebra routines imp lemente d in For tran. C

has lon g been the lingu a franca of programming, so many packages intended for widespread us e exp ort a C-comp atible API, regardless of the langu age of their implementation.

In this sec tion, we’ll bui ld a simple dat a comp ression program that uses cgo, a tool that cre ates Go bindings for C functions. Such tools are cal le d foreign-f unc tion int erfaces (FFIs), and cgo is not the only one for Go programs. SWIG (swig.org) is another ; it prov ides more complex fe atures for int egrat ing wit h C++ class es, but we won’t show it here.

The compress/... su btree of the stand ard librar y prov ides comp ressors and decompressors for popu lar comp ression algor it hms, including LZW (used by the Unix compress command) and DEFL ATE (used by the GNU gzip command). The APIs of these packages vary slig htly in detai ls, but the y al l prov ide a wrapper for an io.Writer that comp resses the dat a wr itt en to it, and a wrapper for an io.Reader that decomp resses the dat a re ad fro m it. For example: package gzip // compress/gzip

func NewWriter(w io.Writer) io.WriteCloser

func NewReader(r io.Reader) (io.ReadCloser, error)

The bzip2 algor it hm, which is bas ed on the elegant Bur rows-Whe eler transfor m, runs slower than gzip but yields sig nificant ly better compression. The compress/bzip2 package provides a decompressor for bzip2, but at the moment the package provides no compressor. Bui lding on e from scratch is a subst ant ial under tak ing, but there is a wel l-do cumente d and hig h-performance open-s ource C implementation, the libbzip2 package fro m bzip.org.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

362

CHAPTER 13. LOW-LEVEL PROGRAMMING

If the C librar y were small, we wou ld just por t it to pure Go, and if its per for mance were not cr itical for our pur pos es, we would be better off inv oking a C program as a helper subprocess using the os/exec package. It’s when you need to use a complex, per for mance-cr itical librar y with a nar row C API that it may make sense to wrap it using cgo. For the rest of this chapt er, we’ll wor k thro ugh an example.

Fr om the libbzip2 C package, we need the bz_stream st ruc t type, which holds the inp ut and output buf fers, and three C functions: BZ2_bzCompressInit, which allocates the stream’s buffers; BZ2_bzCompress, which compresses dat a from the inp ut buf fer to the out put buf fer ; and BZ2_bzCompressEnd, which releases the buf fers. (Don’t wor ry about the mech anics of the libbzip2 package; the pur pos e of this example is to show how the par ts fit toget her.) We’ll cal l the BZ2_bzCompressInit and BZ2_bzCompressEnd C functions direc tly fro m Go, but for BZ2_bzCompress, we’ll define a wrapper function in C, to show how it’s don e. The C

source file below lives alongside the Go code in our package: gopl.io/ch13/bzip

/* This file is gopl.io/ch13/bzip/bzip2.c,

*/

/* a simple wrapper for libbzip2 suitable for cgo. */

#include <bzlib.h>

int bz2compress(bz_stream *s, int action,

ptg16091132

char *in, unsigned *inlen, char *out, unsigned *outlen) {

s->next_in = in;

s->avail_in = *inlen;

s->next_out = out;

s->avail_out = *outlen;

int r = BZ2_bzCompress(s, action);

*inlen -= s->avail_in;

*outlen -= s->avail_out;

return r;

}

No w let’s tur n to the Go code, the firs t part of which is shown below. The import "C" de claration is speci al. There is no package C, but this imp ort causes go build to preprocess the file using the cgo to ol before the Go compi ler sees it.

// Package bzip provides a writer that uses bzip2 compression (bzip.org).

package bzip

/*

#cgo CFLAGS: -I/usr/include

#cgo LDFLAGS: -L/usr/lib -lbz2

#include <bzlib.h>

int bz2compress(bz_stream *s, int action,

char *in, unsigned *inlen, char *out, unsigned *outlen);

*/

import "C"

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 13.4. CALLING C CODE WITH CGO

363

import (

"io"

"unsafe"

)

type writer struct {

w

io.Writer // underlying output stream

stream *C.bz_stream

outbuf [64 * 1024]byte

}

// NewWriter returns a writer for bzip2-compressed streams.

func NewWriter(out io.Writer) io.WriteCloser {

const (

blockSize = 9

verbosity = 0

workFactor = 30

)

w := &writer{w: out, stream: new(C.bz_stream)}

C.BZ2_bzCompressInit(w.stream, blockSize, verbosity, workFactor) return w

}

During preprocessing, cgo generates a temporar y package that contains Go declarat ions cor-ptg16091132

resp onding to all the C functions and typ es us ed by the file, such as C.bz_stream and C.BZ2_bzCompressInit. The cgo to ol dis cov ers these typ es by inv oking the C compi ler in a sp eci al way on the contents of the comment that pre cedes the imp ort declarat ion.

The comment may also contain #cgo direc tives that specif y ext ra opt ion s to the C toolchain.

The CFLAGS and LDFLAGS values cont ribut e ext ra arguments to the compi ler and lin ker command s so that the y can locate the bzlib.h he ader file and the libbz2.a archive librar y. The example assumes that these are ins tal le d bene ath /usr on your system. You may need to alt er or delete these flags for your ins tal lat ion.

NewWriter makes a cal l to the C function BZ2_bzCompressInit to initialize the buf fers for the stream. The writer type includes anot her buf fer that will be used to drain the decompressor’s out put buf fer.

The Write method, shown below, feeds the uncompressed data to the compressor, cal ling the func tion bz2compress in a loop unt i l al l the dat a has been con sumed. Obs erve that the Go prog ram may access C typ es li ke bz_stream, char, and uint, C functions like bz2compress, and even obj e ct-li ke C prepro cessor macros such as BZ_RUN, all through the C.x notation. The C.uint type is distinc t from Go’s uint type, even if bot h have the same width.

func (w *writer) Write(data []byte) (int, error) {

if w.stream == nil {

panic("closed")

}

var total int // uncompressed bytes written

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

364

CHAPTER 13. LOW-LEVEL PROGRAMMING

for len(data) > 0 {

inlen, outlen := C.uint(len(data)), C.uint(cap(w.outbuf)) C.bz2compress(w.stream, C.BZ_RUN,

(*C.char)(unsafe.Pointer(&data[0])), &inlen,

(*C.char)(unsafe.Pointer(&w.outbuf)), &outlen)

total += int(inlen)

data = data[inlen:]

if _, err := w.w.Write(w.outbuf[:outlen]); err != nil {

return total, err

}

}

return total, nil

}

Each iterat ion of the loop passes bz2compress the address and lengt h of the remaining portion of data, and the address and cap acity of w.outbuf. The two lengt h var iables are pass ed by their addresses, not their values, so that the C function can update them to indic ate how much uncompressed dat a was cons ume d and how much compressed dat a was pro duce d.

Each chun k of compressed dat a is then writt en to the underly ing io.Writer.

The Close method has a similar str ucture to Write, using a loop to flush out any remaining comp ressed dat a from the stream’s out put buf fer.

// Close flushes the compressed data and closes the stream.

// It does not close the underlying io.Writer.

ptg16091132

func (w *writer) Close() error {

if w.stream == nil {

panic("closed")

}

defer func() {

C.BZ2_bzCompressEnd(w.stream)

w.stream = nil

}()

for {

inlen, outlen := C.uint(0), C.uint(cap(w.outbuf))

r := C.bz2compress(w.stream, C.BZ_FINISH, nil, &inlen, (*C.char)(unsafe.Pointer(&w.outbuf)), &outlen)

if _, err := w.w.Write(w.outbuf[:outlen]); err != nil {

return err

}

if r == C.BZ_STREAM_END {

return nil

}

}

}

Up on complet ion, Close calls C.BZ2_bzCompressEnd to release the stream buf fers, using defer to ens ure that this happens on all retur n paths. At this point the w.stream pointer is no lon g er safe to dereference. To be defensive , we set it to nil, and add explicit nil che cks to each met hod, so that the program panics if the user mistakenly cal ls a met hod after Close.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

SECTION 13.4. CALLING C CODE WITH CGO

365

No t on ly is writer not con cur rency-s afe, but con cur rent cal ls to Close and Write could cause the program to crash in C code. Fixing this is Exercis e 13.3.

The program below, bzipper, is a bzip2 comp ressor command that uses our new package. It behaves like the bzip2 command present on many Unix systems.

gopl.io/ch13/bzipper

// Bzipper reads input, bzip2-compresses it, and writes it out.

package main

import (

"io"

"log"

"os"

"gopl.io/ch13/bzip"

)

func main() {

w := bzip.NewWriter(os.Stdout)

if _, err := io.Copy(w, os.Stdin); err != nil {

log.Fatalf("bzipper: %v\n", err)

}

if err := w.Close(); err != nil {

log.Fatalf("bzipper: close: %v\n", err)

}

ptg16091132

}

In the session below, we use bzipper to compress /usr/share/dict/words, the system dic-tion ary, fro m 938,848 bytes to 335,405 bytes—ab out a third of its original size—then uncompress it wit h the system bunzip2 command. The SHA256 hash is the same before and after, giving us confidence that the compressor is wor king cor rec tly. (If you don’t have sha256sum on your system, use your solut ion to Exercis e 4.2.)

$ go build gopl.io/ch13/bzipper

$ wc -c < /usr/share/dict/words

938848

$ sha256sum < /usr/share/dict/words

126a4ef38493313edc50b86f90dfdaf7c59ec6c948451eac228f2f3a8ab1a6ed -

$ ./bzipper < /usr/share/dict/words | wc -c

335405

$ ./bzipper < /usr/share/dict/words | bunzip2 | sha256sum 126a4ef38493313edc50b86f90dfdaf7c59ec6c948451eac228f2f3a8ab1a6ed -

We’ve demon strated lin king a C librar y into a Go program. Going in the other direc tion, it’s also possible to compi le a Go program as a static arc hive that can be lin ked int o a C program or as a share d librar y that can be dynamic ally loade d by a C program. We’ve only scratch ed the sur face of cgo here , and there is much more to say about memor y management, point ers, callbacks, signal handling, str ings, errno, finalizers, and the rel ation ship bet ween goroutines and operat ing system threads, much of it ver y su btle. In par tic ular, the rules for cor rec tly passing point ers fro m Go to C or vice versa are complex, for reasons simi lar to those we

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

366

CHAPTER 13. LOW-LEVEL PROGRAMMING

discussed in Sec tion 13.2, and not yet author it ative ly specified. For fur ther reading, start wit h https://golang.org/cmd/cgo.

Exercis e 13.3: Us e sync.Mutex to make bzip2.writer safe for con cur rent use by multiple goro utines.

Exercis e 13.4: Depending on C librar ies has its drawbacks. Provide an alternat ive pure-G o implementation of bzip.NewWriter that uses the os/exec package to run /bin/bzip2 as a su bprocess.

13.5. Another Word of Caution

We ended the pre vious chapt er wit h a war ning about the dow nsides of the reflec tion int erface.

That warning app lies wit h even more force to the unsafe package des crib ed in this chapt er.

High-le vel langu ages insu late programs and programmers not only fro m the arc ane specifics of indiv idu al computer ins truction sets, but fro m dep endence on irrelevancies like where in memory a var iable lives, how big a dat a type is, the det ails of str ucture layout, and a host of ot her implementation det ails. Because of that ins ulating layer, it’s possible to write programs that are safe and robust and that will run on any operat ing system wit hout change .

The unsafe package lets programmers reach through the ins ulation to use some cruci al but ptg16091132

ot her wis e inaccessible feature , or perhaps to achie ve hig her per for mance. The cost is usu ally to por tabilit y and safet y, so one uses unsafe at one’s per il. Our advice on how and when to us e unsafe paral lels Knu th’s comments on premature opt imizat ion, which we quote d in Section 11.5. Most programmers will never need to use unsafe at all. Never theless, there will occasionally be sit uat ions where som e cr itical pie ce of code can be best writt en using unsafe.

If caref ul study and measurement indic ates that unsafe re ally is the best appro ach, restr ict it to as small a reg ion as possible, so that most of the program is obliv iou s to its use.

Fo r now, put the last two chapt ers in the back of your mind. Write som e su bst ant ial Go prog rams. Avoid reflect and unsafe; com e back to these chapt ers only if you must.

Me anw hile, happy Go programming. We hop e you enj oy writing Go as much as we do.

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

Index

!, negat ion operator 63

Abst rac t Sy ntax Not ation One

st ruc t field 104, 105, 106, 162

%, remainder operator 52, 166

(ASN.1) 107

API

&&, short-circuit AND operator 63

abstrac t type 24, 171

encoding 213, 340

&, address-of operator 24, 32, 94,

abstrac tion, premature 216, 316, 317

er ror 127, 152

158, 167

ad hoc poly mor phism 211

package 284, 296, 311, 333, 352

ptg16091132

&, imp licit 158, 167

address of local variable 32, 36

runt ime 324

&^, AND-NOT operator 53

address of str uct lit eral 103

SQL 211

&^, bit-cle ar op erator 53

addressable expression 159, 341

system cal l 196

' quot e ch arac ter 56

addressable value 32

template 115

*, indirec tion operator 24, 32

address-of operator & 24, 32, 94,

token-b ased decoder 213, 215,

++, increment statement 5, 37, 94

158, 167

347

+, str ing con catenat ion operator 5, ag gregate typ e 81, 99

APL programming langu age xiii

65

Alef programming langu age xiii

append built-in function 88, 90, 91

+, unary operator 53

algor it hm

appendInt example 88

+=, -=, etc., assig nment operator 5

breadt h-firs t search 139, 239

argument

-, unary operator 53

depth-firs t search 136

... 139, 142

--, decrement statement 5, 37

Fibonacci 37, 218

command-line 4, 18, 33, 43, 179,

... argument 139, 142

GCD 37

180, 290, 313

... ar ray lengt h 82

insertion sor t 101

func tion 119

... parameter 91, 142, 143, 172

Liss ajous 15

pointer 33, 83

... path 292, 299

slice rot ation 86

slice 86

/*...*/ comment 5, 25

topolog ical sor t 136

ar ithmetic expression evaluator 197

// comment 5, 25

ali asing, point er 33

ar ray

:= short var iable decl arat ion 5, 31,

alig nment 354

comp arison 83

49

al location

lengt h, ... 82

<<, lef t shif t op erator 54

he ap 36

literal 82, 84

==, comparison operator 40, 63

memory 36, 71, 89, 169, 209, 322

type 81

>>, rig ht shif t op erator 54

st ack 36

underly ing 84, 88, 91, 187

^, bit w ise complement operator 53

anch or element, HTML 122

zero value 82

^, exc lusive OR operator 53

AND operator &&, short-circuit 63

ASCII 56, 64, 66, 67, 305

_, blank identifier 7, 38, 95, 120, 126,

AND-NOT operator & 53

ASN.1 (Abstrac t Sy ntax Not ation

287

animat ion, GIF 13

One) 107

` backquot e ch arac ter 66

anonymou s

assemb l y line, cake 234

| in templ ate 113

func tion 22, 135, 236

assertion

|, bit w ise OR operator 166, 167

func tion, defer 146

func tion 316

||, short-circuit OR operator 63

func tion, rec ursive 137

interface typ e 208, 210

367

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

368

INDEX

test 306

break st atement 24, 46

C programming langu age xii, xv, 1,

type 205, 211

break st atement, lab ele d 249

6, 52, 260, 361

assig nabilit y 38, 175

br ittle test 317

cach e, conc urrent non-blo cking 272

assig nabilit y, int erface 175

broadc ast 251, 254, 276

cach e, non-blo cking 275

assig nment

Brooks, Fre d xiv

cake ass emb l y line 234

implicit 38

btoi func tion 64

call

mu ltiple-value 37

buffered channel 226, 231

by reference 83

op erator +=, -=, etc. 5

bufio package 9

by value 83, 120, 158

op erator s 36, 52

bufio.NewReader func tion 98

interface met hod 182

st atement 5, 7, 36, 52, 94, 173

bufio.NewScanner func tion 9

ok value fro m func tion 128

tuple 31, 37

(*bufio.Reader).ReadRune

calling C fro m Go 361

asso ciativity, operator 52

method 98

camel cas e 28

atomic operat ion 264

bufio.Scanner type 9

cancel lat ion 251, 252

attack, HTML inj e ction 115

(*bufio.Scanner).Err method 97

cancel lat ion of HTTP request 253

attack, SQL inj e ction 211

(*bufio.Scanner).Scan method 9

cap built-in function 84, 232

autoescape example 117

(*bufio.Scanner).Split method

capacity, channel 226, 232, 233

99

capacity, slice 88, 89

back-do or, package 315

bufio.ScanWords func tion 99

capt uring iterat ion var iable 140

back-off, exp onent ial 130

+build comments 296

capt uring loop var iable 141, 236,

backquot e ch arac ter, ` 66

build con straints 296

240

bank example package 258, 261, 263

build tags 296

case in typ e switch 212

bare return 126

building packages 293

case, select 245

basename example 72

built-in function

Celsius type 39

behavior, undefine d 260

append 88, 90, 91

CelsiusFlag func tion 181

Benchmark func tion 302, 321

cap 84, 232

cf example 43

bidirec tion al to unidirec tion al

close 226, 228, 251

cgo to ol 361, 362

ch annel conversion 231

complex 61

<-ch, channel receive 18, 225, 232

binary

copy 89

ch<-, channel send 18, 225, 232

op erator s, table of 52

delete 94

ch aining, met hod 114

ptg16091132

semaph ore 262

imag 61

chan type 225

tree 102

len 4, 54, 64, 65, 81, 84, 233

ch annel

bit vec tor 165

make 9, 18, 88, 94, 225

buffered 226, 231

bit-cle ar op erator & 53

new 34

capacity 226, 232, 233

bit-s et data typ e 77

panic 148, 149

close 228, 251

bit w ise

real 61

closing a 225

comp lement operator ^ 53

recover 152

communic ation 225, 245

op erator s, table of 53

built-in interface, error 196

comp arison 225

OR operator | 166, 167

built-in typ e, error 11, 128, 149,

conv ersion, bidirec tion al to

bl ack-b ox test 310

196

unidirec tion al 231

bl ank identifier _ 7, 38, 95, 120, 126,

byte slice to str ing conversion 73

draining a 229, 252

287

byte type 52

make 18, 225

bl ank imp ort 287

ByteCounter example 173

ni l 246, 249

block

bytes package 71, 73

polling 246

file 46

bytes.Buffer type 74, 169, 172, 185

range ov er 229

lexic al 46, 120, 135, 141, 212

(*bytes.Buffer).Grow method

re ceive <-ch 18, 225, 232

lo cal 46

169

re ceive , non-blo cking 246

package 46

(*bytes.Buffer).WriteByte

re ceive , ok value fro m 229

univers e 46

method 74

send ch<- 18, 225, 232

blocking profile 324

(*bytes.Buffer).WriteRune

sy nchro nou s 226

Blog , Go xvi, 326

method 74

type 18

boiling example 29

(*bytes.Buffer).WriteString

type <-chan T, receive-only 230

bool type 63

method 74

type chan<- T, send-only 230

boole an

bytes.Equal func tion 86

type, unidirec tion al 230, 231

cons tant, false 63

bzip C code 362

unbuffered 226

cons tant, true 63

bzip example package 363

zero value 225, 246

zero value 30

bzipper example 365

ch arac ter conv ersion 71

breadthFirst func tion 139

ch arac ter test 71

breadt h-firs t search algor it hm 139,

C++ programming langu age xiv, xv,

charcount example 98

239

361

chat example 254

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

INDEX

369

ch at ser ver 253

safe 275

cyclic dat a st ruc ture 337

CheckQuota func tion 312, 313

safety 256, 257, 272, 365

cyclic test dep endency 314

client, emai l 312

with share d var iables 257

client, SMTP 312

conc urrent

data

clock example 220, 222

clock ser ver 219

race 259, 267, 275

clock ser ver, con cur rent 219

direc tor y traversal 247

st ruc ture, cyc lic 337

close built-in function 226, 228,

echo ser ver 222

st ruc ture, rec ursive 101, 102, 107

251

non-blo cking cache 272

type, bit-s et 77

close, channel 228, 251

we b craw ler 239

database driver, MySQL 284

clos er goro utine 238, 250

confinement, ser ial 262

database/sql package 211, 288

closing a channel 225

confinement, var iable 261

daysAgo func tion 114

closure, lexic al 136

consistenc y, sequential 268, 269

deadbeef 55, 80

cmplx.Sqrt func tion 61

const de clarat ion 14, 75

de adlo ck 233, 240, 265

co de

cons tant

de clarat ion

format 3, 6, 9, 48

false boole an 63

const 14, 75

point, Unico de 67

generator, iota xiii, 77

func 3, 29, 119

produc tion 301

time.Minute 76

import 3, 28, 42, 284, 285, 362

ColoredPoint example 161

time.Second 164

method 40, 155

comma example 73

true boole an 63

package 2, 28, 41, 285

command, test ing a 308

types, untyped 78

package-le vel 28

command-line argument 4, 18, 33,

cons tants, pre cision of 78

scop e 45, 137

43, 179, 180, 290, 313

cons traints, bui ld 296

sh adowing 46, 49, 206, 212

comment

cont ent ion, lock 267, 272

short var iable 5, 7, 30, 31

/*...*/ 5, 25

cont ext swit c h 280

st atement, short var iable 7

// 5, 25

continue st atement 24, 46

struct 99

do c 42, 296

continue st atement, lab ele d 249

type 39

// Output 326

cont rac ts, interfaces as 171

var 5, 30

comments, +build 296

cont rol flow 46

de clarat ions, order of 48

communic ating sequential pro cesses

conv ersion

de code example, S-expression 347

ptg16091132

(CSP) xiii, 217

bidirec tion al to unidirec tion al

de coder API, token-b ased 213, 215,

communic ation, channel 225, 245

ch annel 231

347

comp arabilit y 9, 38, 40, 53, 86, 93,

byte slice to str ing 73

de coding, S-expression 344

97, 104

ch arac ter 71

de coding, XML 213

comp arison

implicit 79

de crement statement -- 5, 37

ar ray 83

narrow ing 40, 55

dedup example 97

ch annel 225

numeric 79

de ep equivalence 87, 317, 358

func tion 133

op erat ion 40, 55, 64, 71, 78, 79,

default case in select 246

interface 184

173, 187, 194, 208, 231, 353, 358

default case in switch 23

map 96

rune slice to str ing 71

default case in typ e switch 212

op erator == 40, 63

rune to str ing 71

defer anonymou s func tion 146

op erator s 40, 93

st ring 71

defer example 150, 151

op erator s, table of 53

st ring to byte slice 40, 73

defer st atement 144, 150, 264

slice 87

st ring to rune slice 71, 88

defer red function cal l 144

st ring 65

unsafe.Pointer 356

delete built-in function 94

st ruc t 104

copy built-in function 89

depth-firs t search algor it hm 136

comp ilation, sep arate 284

countdown example 244, 245, 246

dereference, imp licit 159

comp lement operator ^, bit w ise 53

counting semaphore 241

di agram

complex built-in function 61

coverage, statement 318, 320

helloworld su bst ring 69

complex type 61

coverage, test 318

pip eline 228

comp osite lit eral 14

coverage_test example 319

slice cap acity growt h 90

comp osite typ e xv, 14, 81

CPU profile 324

slice of months 84

comp osition, paral lel 224

crawl example 240, 242, 243

st ring sharing 65

comp osition, typ e xv, 107, 162, 189

craw ler, con cur rent web 239

st ruc t hole 355

compress/bzip2 package 361

craw ler, web 119

thumbnail sequence 238

comp ression 361

cr itical sec tion 263, 270, 275

dig ital artifac t example 178

conceptu al int egr ity xiv

cross-comp ilation 295

Dijkstra, Edsger 318

concrete typ e 24, 171, 211, 214

cr yptography 55, 83, 121, 325

Di lbert 100

conc urrency 17, 217, 257

crypto/sha256 package 83

direc ted acyclic graph 136, 284

excessive 241, 242

customSort example 190

direc tor y traversal, con cur rent 247

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

370

INDEX

discriminated union 211, 213, 214

errors package 196

nonempty 92

Display func tion 333

errors.New func tion 196

outline 123, 133

display example 333

es cap e

package, bank 258, 261, 263

display func tion 334

hexade cimal 66

package, bzip 363

displaying met hods of a typ e 351

HTML 116

package, format 332

Distance func tion 156

octal 66

package, geometry 156

do c comment 42, 296

sequence 10

package, http 192

doc.go do c comment file 42, 296

sequences, table of 66

package, links 138

do cumentation, package 296

Unico de 68, 107

package, memo 273

domain name, imp ort pat h 284

URL 111

package, params 348

dot . in templ ate 113

es caping var iables 36

package, storage 312, 313

down loading packages 292

eval example 198

package, tempconv 42

Dr. Strange lov e 336

event multiplexing 244

package, thumbnail 235

draining a channel 229, 252

events 227, 244

palindrom e 303, 305, 308

du example 247, 249, 250

Example func tion 302, 326

params 348

dup example 9, 11, 12

example

Parse 152

dup lic ate sup pression 276

autoescape 117

pipeline 228, 230, 231

dy namic disp atch 183

basename 72

pl aylist 187

dy namic typ e, int erface 181

boiling 29

rev 86

ByteCounter 173

reverb 223, 224

echo example 5, 7, 34, 309

bzipper 365

server 19, 21

echo test 309

cf 43

sexpr 340

echo ser ver, con cur rent 222

charcount 98

S-expression decode 347

echo_test.go 310

chat 254

sha256 83

ef fec tive tests, writing 316, 317

clock 220, 222

sleep 179

emai l client 312

ColoredPoint 161

spinner 218

embarrassingly paral lel 235

comma 73

squares 135

embedde d st ruc t field 161

countdown 244, 245, 246

sum 142

embedding, int erface 174

coverage_test 319

surface 59, 203

ptg16091132

embedding, str uct 104, 161

crawl 240, 242, 243

tempconv 39, 180, 289

Employee st ruc t 100

customSort 190

temp erature conversion 29

empt y

dedup 97

tempflag 181

interface typ e 176

defer 150, 151

test of word 303

select st atement 245

dig ital artifac t 178

thumbnail 236, 237, 238

st ring 5, 7, 30

display 333

title 153

st ruc t 102

du 247, 249, 250

topoSort 136

encapsulat ion 168, 284

dup 9, 11, 12

trace 146

encoding API 213, 340

echo 5, 7, 34, 309

treesort 102

enco ding, S-expression 338

eval 198

urlvalues 160

encoding/json package 107

fetch 16, 148

wait 130

encoding/xml package 107, 213

fetchall 18

word 303, 305, 308

end of file (EOF) 131

findlinks 122, 125, 139

xmlselect 215

enum 77

ftoc 29

appendInt 88

enviro nment var iable

github 110, 111

exception 128, 149

GOARCH 292, 295

graph 99

excessive con cur rency 241, 242

GOMAXPROCS 281, 321

helloworld 1, 2

exclusion, mut ual 262, 267

GOOS 292, 295

http 192, 194, 195

exclusive lock 263, 266, 270

GOPATH xv i, 291, 295

intset 166

exclusive OR operator ^ 53

GOROOT 292

issues 112

exp onent ial back-off 130

equal func tion 87, 96

issueshtml 115

exp ort of str uct field 101, 106, 109,

equality, point er 32

issuesreport 114

110, 168

equivalence, deep 87, 317, 358

jpeg 287

export_test.go file 315

error built-in interface 196

lissajous 14, 22, 35

Expr.Check method 202

error built-in typ e 11, 128, 149, 196

mandelbrot 62

expression

er ror API 127, 152

memo 275, 276, 277, 278, 279

addressable 159, 341

error.Error method 196

methods 351

evaluator 197

errorf func tion 143

movie 108, 110

method 164

er ror-handling strateg ies 128, 152,

netcat 221, 223, 227

re ceive 225

310, 316

netflag 78

Expr.Eval method 199

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

INDEX

371

extending a slice 86

floating-p oint

flag.String 34

Extensible Marku p Language (XML)

numb er 56

fmt.Errorf 129, 196

107

precision 56, 57, 63, 78

fmt.Fprintf 172

exter nal test package 285, 314

tr unc ation 40, 55

fmt.Printf 10

fmt package 2

fmt.Println 2

Fahrenheit type 39

fmt.Errorf func tion 129, 196

fmt.Scanf 75

fai lure message , test 306

fmt.Fprintf func tion 172

fmt.Sscanf 180

fallthrough st atement 23, 212

fmt.Printf func tion 10

forEachNode 133

false boole an cons tant 63

fmt.Println func tion 2

formatAtom 332

fetch example 16, 148

fmt.Scanf func tion 75

gcd 37

fetchall example 18

fmt.Sscanf func tion 180

hand ler 19, 21, 152, 191, 194, 195,

fib func tion 37, 218

fmt.Stringer interface 180, 210

348

Fibonacci algor it hm 37, 218

for scop e 47

html.Parse 121, 125

field

for st atement 6

http.DefaultServeMux 195

anonymou s st ruc t 104, 105, 106,

forEachNode func tion 133

http.Error 193

162

foreig n-function int erface (FFI) 361

http.Get 16, 18

embedde d st ruc t 161

format, code 3, 6, 9, 48

http.Handle 195

exp ort of str uct 101, 106, 109,

format example package 332

http.HandleFunc 19, 22, 195

110, 168

formatAtom func tion 332

http.ListenAndServe 19, 191

order, str uct 101, 355

framewor k, web 193

http.NewRequest 253

sele ctor 156

ftoc example 29

http.ServeMux 193

st ruc t 15, 99

func de clarat ion 3, 29, 119

hypot 120

tag, omitempty 109

func tion

imag built-in 61

tag, str uct 109, 348

anonymou s 22, 135, 236

image.Decode 288

figure

append built-in 88, 90, 91

image.RegisterFormat 288

Liss ajous 13

argument 119

incr 33

Mandelbro t 63

assertion 316

init 44, 49

3-D sur face 58, 203

Benchmark 302, 321

intsToString 74

Fi le Transfer Pro tocol (FTP) 222

body, missing 121

io.Copy 17, 18

ptg16091132

file

breadthFirst 139

ioutil.ReadAll 16, 272

block 46

btoi 64

ioutil.ReadDir 247

export_test.go 315

bufio.NewReader 98

ioutil.ReadFile 12, 145

name, Micros oft Windows 72

bufio.NewScanner 9

io.WriteString 209

name, POSIX 72

bufio.ScanWords 99

itob 64

_test.go 285, 302, 303

bytes.Equal 86

json.Marshal 108

findlinks example 122, 125, 139

call, defer red 144

json.MarshalIndent 108

fixe d-size stack 124

call, ok value fro m 128

json.NewDecoder 111

flag package 33, 179

cap built-in 84, 232

json.NewEncoder 111

flag

CelsiusFlag 181

json.Unmarshal 110, 114

go tool -bench 321

CheckQuota 312, 313

len built-in 4, 54, 64, 65, 81, 84,

go tool -benchmem 322

close built-in 226, 228, 251

233

go tool -covermode 319

cmplx.Sqrt 61

links.Extract 138

go tool -coverprofile 319

comp arison 133

literal 22, 135, 227

go tool -cpuprofile 324

complex built-in 61

log.Fatalf 49, 130

go tool -nodecount 325

copy built-in 89

main 2, 310

go tool -text 325

daysAgo 114

make built-in 9, 18, 88, 94, 225

go tool -web 326

delete built-in 94

math.Hypot 156

godoc -analysis 176

Display 333

math.Inf 57

go list -f 315

display 334

math.IsInf 57

go -race 271

Distance 156

math.IsNaN 57

go test -race 274

equal 87, 96

math.NaN 57

go test -run 305

errorf 143

mu lti-value d 11, 30, 37, 96, 125,

go test -v 304

errors.New 196

126

flag.Args func tion 34

Example 302, 326

mustCopy 221

flag.Bool func tion 34

fib 37, 218

net.Dial 220

flag.Duration func tion 179

flag.Args 34

net.Listen 220

flag.Parse func tion 34

flag.Bool 34

new built-in 34

flag.String func tion 34

flag.Duration 179

ni l 132

flag.Value interface 179, 180

flag.Parse 34

os.Close 11

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

372

INDEX

os.Exit 16, 34, 48

title 144, 145

goimports to ol 3, 44, 286

os.Getwd 48

type 119, 120

go install 295

os.IsExist 207

unicode.IsDigit 71

golang.org/x/net/html package

os.IsNotExist 207

unicode.IsLetter 71

122

os.IsPermission 207

unicode.IsLower 71

golint to ol 292

os.Open 11

unicode.IsSpace 93

go list 298, 315

os.Stat 247

unicode.IsUpper 71

go list -f flag 315

panic built-in 148, 149

unsafe.AlignOf 355

GOMAXPROCS enviro nment var iable

parameter 119

unsafe.Offsetof 355

281, 321

params.Unpack 349

unsafe.Sizeof 354

GOOS enviro nment var iable 292, 295

png.Encode 62

url.QueryEscape 111

GOPATH enviro nment var iable xvi,

PopCount 45

utf8.DecodeRuneInString 69

291, 295

real built-in 61

utf8.RuneCountInString 69

gopl.io repository xvi

recover built-in 152

value 132

go -race flag 271

re cursive anony mou s 137

var iadic 142, 172

GOROOT enviro nment var iable 292

reflect.TypeOf 330

visit 122

goro utine 18, 217, 233, 235

reflect.ValueOf 331, 337

WaitForServer 130

clos er 238, 250

reflect.Zero 345

walkDir 247

identity 282

regexp.Compile 149

zero value 132

le ak 233, 236, 246

regexp.MustCompile 149

monitor 261, 277

resu lt list 119

garb age col lec tion xi, xiii, 7, 35, 230,

mu ltiplexing 281

runtime.Stack 151

353, 357

vs. OS thread 280

SearchIssues 111

garb age col lec tor, mov ing 357

go run 2, 294

sexpr.Marshal 340

GCD algor it hm 37

go test 301, 302, 304

sexpr.readList 347

gcd func tion 37

go test -race flag 274

sexpr.Unmarshal 347

geometry example package 156

go test -run flag 305

sig nature 120

geometry.Point.Distance method

go test -v flag 304

sort.Float64s 191

156

goto st atement 24

sort.Ints 191

gett er met hod 169

graph example 99

ptg16091132

sort.IntsAreSorted 191

GIF animat ion 13

GraphViz 326

sort.Reverse 189

GitHub issue tracker 110

Gr ies emer, Rob ert xi

sort.Strings 95, 137, 191

github example 110, 111

growth, stack 124, 280, 358

Sprint 330

Go

gu arding mut ex 263

sqlQuote 211, 212

Pl ayg round xvi, 326

strconv.Atoi 22, 75

Blog xvi, 326

half-open int erval 4

strconv.FormatInt 75

issue 110, 112, 358

hand ler function 19, 21, 152, 191,

strconv.Itoa 75

go to ol 2, 42, 44, 290

194, 195, 348

strconv.ParseInt 75

go tool -bench flag 321

‘‘happ ens before’’ re lat ion 226, 257,

strconv.ParseUint 75

go tool -benchmem flag 322

261, 277

strings.Contains 69

go tool -covermode flag 319

‘‘has a’’ re lat ions hip 162

strings.HasPrefix 69

go tool -coverprofile flag 319

hash table 9, 93

strings.HasSuffix 69

go tool -cpuprofile flag 324

Haskel l prog ramming langu age xiv

strings.Index 289

go tool -nodecount flag 325

he ap

strings.Join 7, 12

go tool pprof 325

al location 36

strings.Map 133

go tool -text flag 325

profile 324

strings.NewReader 289

go tool -web flag 326

var iable 36

strings.NewReplacer 289

go tool cover 318, 319

helloworld example 1, 2

strings.Split 12

go doc to ol 25

helloworld su bst ring diagram 69

strings.ToLower 72

go st atement 18, 218

hexade cimal escape 66

strings.ToUpper 72

GOARCH enviro nment var iable 292,

hexade cimal literal 55

template.Must 114

295

hidden point er 357

template.New 114

go build 2, 286, 293, 294

Ho are , To ny xiii

Test 302

go doc 296

hole, str uct 354

time.After 245

godoc -analysis flag 176

HTML

time.AfterFunc 164

godoc to ol xvi, 25, 297, 326

anch or element 122

time.Now 220

go env 292

es cap e 116

time.Parse 220

gofmt to ol 3, 4, 44, 286

injec tion att ack 115

time.Since 114

go get xv i, 2, 292, 293

metach arac ter 116

time.Tick 244, 246

go help 290

pars er 121

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

INDEX

373

html.Parse func tion 121, 125

285, 362

interfaces as cont rac ts 171

html/template package 113, 115

import

internal package 298

HT TP

bl ank 287

intset example 166

GET request 21, 127, 272, 348

path 284

intsToString func tion 74

POST request 348

path dom ain name 284

invar iants 159, 169, 170, 265, 284,

re quest, cancellat ion of 253

renaming 286

311, 352

re quest multiplexer 193

incr func tion 33

io package 174

http example 192, 194, 195

increment statement ++ 5, 37, 94

io.Closer interface 174

http example package 192

index operat ion, str ing 64

io.Copy func tion 17, 18

(*http.Client).Do method 253

indirec tion operator * 24, 32

io.Discard st ream 22

http.DefaultClient var iable 253

infinite loop 6, 120, 228

io.Discard var iable 18

http.DefaultServeMux func tion

infor mat ion hiding 168, 284

io.EOF var iable 132

195

init func tion 44, 49

io/ioutil package 16, 145

http.Error func tion 193

initializat ion

io.Reader interface 174

http.Get func tion 16, 18

lazy 268

iota cons tant generator xiii, 77

http.Handle func tion 195

package 44

ioutil.ReadAll func tion 16, 272

http.HandleFunc func tion 19, 22,

st atement in if 22, 206

ioutil.ReadDir func tion 247

195

st atement in switch 24

ioutil.ReadFile func tion 12, 145

http.Handler interface 191, 193

initializer list 30

io.Writer interface 15, 22, 172,

http.HandlerFunc type 194, 203

injec tion att ack, HTML 115

174, 186, 208, 209, 309

http.ListenAndServe func tion 19,

injec tion att ack, SQL 211

io.WriteString func tion 209

191

in-place slice techniques 91

‘‘is a’’ re lat ions hip 162, 175

http.NewRequest func tion 253

insertion sor t algor it hm 101

issue, Go 110, 112, 358

http.Request type 21, 253

int type 52

issue tracker, GitHu b 110

(*http.Request).ParseForm

integer

issues example 112

method 22, 348

literal 55

issueshtml example 115

http.ResponseWriter type 19, 22,

ov erflow 53, 113

issuesreport example 114

191, 193

sig ned 52, 54

it erat ion order, map 95

http.ServeMux func tion 193

unsig ned 52, 54

it erat ion var iable, capturing 140

ptg16091132

hypot func tion 120

integrat ion test 314

itob func tion 64

interface

identifier _, blank 7, 38, 95, 120, 126,

assig nabilit y 175

Java programming langu age xv

287

comp arison 184

JavaS crip t Objec t No tat ion (JSON)

identifier, qualified 41, 43

dy namic typ e 181

107, 338

identity, goroutine 282

embedding 174

JavaS crip t prog ramming langu age

IEEE 754 stand ard 56, 57

error built-in 196

xv, 107

if, initializat ion statement in 22,

flag.Value 179, 180

jpeg example 287

206

fmt.Stringer 180, 210

JSON

if-else scop e 47

http.Handler 191, 193

interface 110

if-else st atement 9, 22, 47

io.Closer 174

interface, Open Mov ie Dat abas e

imag built-in function 61

io.Reader 174

113

image manipu lat ion 121

io.Writer 15, 22, 172, 174, 186,

interface, xkcd 113

image package 62, 287

208, 209, 309

mars haling 108

image/color package 14

JSON 110

unmarsh aling 110

image.Decode func tion 288

method cal l 182

json.Decoder type 111

image/png package 288

ni l 182

json.Encoder type 111

image.RegisterFormat func tion

pit fal l 184

json.Marshal func tion 108

288

ReadWriteCloser 174

json.MarshalIndent func tion 108

imag inar y literal 61

ReadWriter 174

json.NewDecoder func tion 111

immut abi lit y 261

satisfac tion 171, 175

json.NewEncoder func tion 111

immut abi lit y, str ing 65, 73

sort.Interface 186

json.Unmarshal func tion 110, 114

implementation wit h slice, stack 92,

type 171, 174

215

interface{} type 143, 176, 331

ke yword , type 212

implicit

interface

ke yword s, table of 27

& 158, 167

type ass ertion 208, 210

Knut h, Don ald 323

assig nment 38

type, emp ty 176

conv ersion 79

value 181

label scope 46

dereference 159

with nil point er 184

label, statement 46

import de clarat ion 3, 28, 42, 284,

zero value 182

labele d

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

374

INDEX

break st atement 249

main, package 2, 285, 310

gett er 169

continue st atement 249

make built-in function 9, 18, 88, 94,

(*http.Client).Do 253

st atement 46

225

(*http.Request).ParseForm 22,

layout, memor y 354, 355

make ch annel 18, 225

348

lazy initializat ion 268

make map 9, 18, 94

name 156

le ak, goro utine 233, 236, 246

make slice 88, 322

net.Conn.Close 220

lef t shif t op erator << 54

Mandelbro t figure 63

net.Listener.Accept 220

len built-in function 4, 54, 64, 65,

Mandelbro t set 61

(*os.File).Write 183

81, 84, 233

mandelbrot example 62

path.Distance 157

lexic al block 46, 120, 135, 141, 212

map

prom otion 161

lexic al closure 136

as set 96, 202

re ceiver name 157

lifet ime, var iable 35, 46, 135

comp arison 96

re ceiver parameter 156

links example package 138

element, non existent 94, 95

re ceiver typ e 157

links.Extract func tion 138

it erat ion order 95

reflect.Type.Field 348

Lisp programming langu age 338

literal 94

reflect.Value.Addr 342

Liss ajous algor it hm 15

lo oku p m[key] 94

reflect.Value.CanAddr 342

Liss ajous figure 13

lo oku p, ok value fro m 96

reflect.Value.Interface 331,

lissajous example 14, 22, 35

make 9, 18, 94

342

list, initializer 30

ni l 95

reflect.Value.Kind 332

literal

range ov er 94

sele ctor 156

ar ray 82, 84

type 9, 93

sett er 169

comp osite 14

with slice key 97

String 40, 166, 329

func tion 22, 135, 227

zero value 95

(*sync.Mutex).Lock 21, 146, 263

hexade cimal 55

mars haling JSON 108

(*sync.Mutex).Unlock 21, 146,

imag inar y 61

math package 14, 56

263

integer 55

math/big package 63

(*sync.Once).Do 270

map 94

math/cmplx package 61

(*sync.RWMutex).RLock 266

octal 55

math.Hypot func tion 156

(*sync.RWMutex).RUnlock 266

raw str ing 66

math.Inf func tion 57

(*sync.WaitGroup).Add 238

ptg16091132

rune 56

math.IsInf func tion 57

(*sync.WaitGroup).Done 238

slice 38, 86

math.IsNaN func tion 57

template.Funcs 114

st ring 65

math.NaN func tion 57

template.Parse 114

st ruc t 15, 102, 106

math/rand package 285, 308

(*testing.T).Errorf 200, 304,

lo cal

memo example 275, 276, 277, 278,

306

block 46

279

(*testing.T).Fatal 306

var iable 29, 141

memo example package 273

time.Time.Format 220

var iable, address of 32, 36

memoizat ion 272

value 164

var iable scope 135

memory allocat ion 36, 71, 89, 169,

(*xml.Decoder).Token 213

lo cat ing packages 291

209, 322

methods example 351

lo ck

memory layout 354, 355

methods of a typ e, displ aying 351

cont ent ion 267, 272

metach arac ter, HTML 116

Micros oft Windows file name 72

exclusive 263, 266, 270

method

missing function body 121

mu tex 102, 263, 264, 324

(*bufio.Reader).ReadRune 98

m[key], map lookup 94

non-reent rant 265

(*bufio.Scanner).Err 97

mobi le platfor ms 121

re aders 266

(*bufio.Scanner).Scan 9

Mo dula-2 programming langu age

sh are d 266

(*bufio.Scanner).Split 99

xiii

wr iter 266

(*bytes.Buffer).Grow 169

mo dular ity 283

log package 49, 130, 170

(*bytes.Buffer).WriteByte 74

monitor 264, 275

log.Fatalf func tion 49, 130

(*bytes.Buffer).WriteRune 74

monitor goroutine 261, 277

lo oku p m[key], map 94

(*bytes.Buffer).WriteString

movie example 108, 110

lo oku p, ok value fro m map 96

74

moving garb age col lec tor 357

lo op

call, int erface 182

mu ltimap 160, 193

infinite 6, 120, 228

ch aining 114

mu ltiple-value assig nment 37

range 6, 9

de clarat ion 40, 155

mu ltiplexer, HTTP request 193

var iable, capturing 141, 236, 240

error.Error 196

mu ltiplexing, event 244

var iable scope 141, 236

Expr.Check 202

mu ltiplexing, goroutine 281

while 6

expression 164

mu ltithre ading, share d-memor y

Expr.Eval 199

217, 257

main func tion 2, 310

geometry.Point.Distance 156

mu lti-value d func tion 11, 30, 37, 96,

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

INDEX

375

125, 126

numb er zero value 5, 30

OS thread vs. goroutine 280

mustCopy func tion 221

numeric

os package 4, 206

mu tex 145, 163, 256, 269

conv ersion 79

os.Args var iable 4

gu arding 263

precision 55, 78

os.Close func tion 11

lo ck 102, 263, 264, 324

type 51

os.Exit func tion 16, 34, 48

re ad/w rit e 266, 267

*os.File type 11, 13, 172, 175, 185,

mu tual exc lusion 262, 267

Ob ero n prog ramming langu age xiii

336

MySQL dat abas e dr iver 284

objec t 156

os.FileInfo type 247

objec t-oriente d prog ramming

(*os.File).Write method 183

name

(O OP) 155, 168

os.Getwd func tion 48

method 156

octal escap e 66

os.IsExist func tion 207

method receiver 157

octal lit eral 55

os.IsNotExist func tion 207

package 28, 43

ok value 37

os.IsPermission func tion 207

parameter 120

ok value fro m ch annel receive 229

os.LinkError type 207

sp ace 41, 156, 283

ok value fro m func tion cal l 128

os.Open func tion 11

name d

ok value fro m map looku p 96

os.PathError type 207

resu lt 120, 126

ok value fro m type ass ertion 206

os.Stat func tion 247

resu lt zero value 120, 127

omitempty field tag 109

outline example 123, 133

type 24, 39, 40, 105, 157

Op en Mov ie Dat abas e JSON

// Output comment 326

naming convent ion 28, 169, 174,

interface 113

ov erflow, int eger 53, 113

289

op erat ion, atomic 264

ov erflow, stack 124

naming, package 289

op erat ion, conv ersion 40, 55, 64, 71,

NaN (not a number) 57, 93

78, 79, 173, 187, 194, 208, 231,

package de clarat ion 2, 28, 41, 285

narrow ing conversion 40, 55

353, 358

package

negat ion operator ! 63

op erator

API 284, 296, 311, 333, 352

net package 219

+=, -=, etc., assig nment 5

back-do or 315

netcat example 221, 223, 227

&, address-of 24, 32, 94, 158, 167

bank example 258, 261, 263

net.Conn type 220

&^, AND-NOT 53

block 46

net.Conn.Close method 220

&^, bit-cle ar 53

bufio 9

ptg16091132

net.Dial func tion 220

^, bit w ise complement 53

bytes 71, 73

netflag example 78

|, bit w ise OR 166, 167

bzip example 363

net/http package 16, 191

==, comparison 40, 63

compress/bzip2 361

net.Listen func tion 220

^, exc lusive OR 53

crypto/sha256 83

net.Listener type 220

*, indirec tion 24, 32

database/sql 211, 288

net.Listener.Accept method 220

<<, lef t shif t 54

do cumentation 296

net/smtp package 312

!, negat ion 63

encoding/json 107

net/url package 160

%, remainder 52, 166

encoding/xml 107, 213

networ king 121, 219

>>, rig ht shif t 54

errors 196

new built-in function 34

&&, short-circuit AND 63

exter nal test 285, 314

new, redefining 35

||, short-circuit OR 63

flag 33, 179

ni l

+, str ing con catenat ion 5, 65

fmt 2

ch annel 246, 249

-, unary 53

format example 332

func tion 132

+, unary 53

geometry example 156

interface 182

asso ciativity 52

golang.org/x/net/html 122

map 95

precedence 52, 63

html/template 113, 115

pointer 32

s[i:j], slice 84, 86

http example 192

pointer, int erface wit h 184

s[i:j], subst ring 65, 86

image 62, 287

re ceiver 159, 185

op erator s

image/color 14

slice 87

assig nment 36, 52

image/png 288

non-blo cking

comp arison 40, 93

initializat ion 44

cach e 275

table of binar y 52

internal 298

cach e, conc urrent 272

table of bit w ise 53

io 174

ch annel receive 246

table of comparison 53

io/ioutil 16, 145

select 246

opt imizat ion 264, 321, 323

links example 138

nonempty example 92

opt imizat ion, premature 324

log 49, 130, 170

nonexistent map element 94, 95

OR operator ||, short-circuit 63

main 2, 285, 310

non-reent rant lock 265

order of declarat ions 48

math 14, 56

non-stand ard package 121

order, str uct field 101, 355

math/big 63

numb er, floating-p oint 56

or ganizat ion, wor ksp ace 291

math/cmplx 61

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

376

INDEX

math/rand 285, 308

pars er, HTML 121

Printf %#x 56

memo example 273

Pascal programming langu age xiii

Printf %x 10, 55, 83

name 28, 43

path, ... 292, 299

produc tion code 301

naming 289

path package 72

profile

net 219

path.Distance method 157

blocking 324

net/http 16, 191

path/filepath package 72

CPU 324

net/smtp 312

Pi ke, Rob xi, xiii, 67, 107

he ap 324

net/url 160

pipeline example 228, 230, 231

profiling 324

non-stand ard 121

pip eline 227

prog ramming langu age

os 4, 206

pip eline diagram 228

Alef xiii

params example 348

pit fal l, int erface 184

APL xiii

path 72

pit fal l, scope 140

C++ xiv, xv, 361

path/filepath 72

pl atfor ms, mobile 121

C xii, xv, 1, 6, 52, 260, 361

reflect 330

Pl ayg round, Go xvi, 326

Haskel l xiv

regexp 149

pl aylist example 187

Java xv

runtime 151

png.Encode func tion 62

JavaS crip t xv, 107

sort 95, 186, 189

pointer 24, 32, 34

Lisp 338

storage example 312, 313

ali asing 33

Mo dula-2 xiii

strconv 22, 71, 75

argument 33, 83

Ob ero n xiii

strings 7, 71, 72, 289

equality 32

Pascal xiii

sync 237, 263

hidden 357

Python xv, 193

syscall 196, 208

ni l 32

Ru by xv, 193

tempconv example 42

re ceiver 158, 167

Scheme xiii

testing 285, 302

to str uct 100, 103

Sque ak, Ne wsque ak xiii

text/scanner 344

zero value 32

prom otion, met hod 161

text/tabwriter 188

polling channel 246

prot ocol buf fers 107

text/template 113, 300

poly mor phism, ad hoc 211

Python prog ramming langu age xv,

thumbnail example 235

poly mor phism, subtype 211

193

time 18, 77, 183

PopCount func tion 45

ptg16091132

unicode 71

Po rtable Net wor k Graphics (PNG)

qu alified identifier 41, 43

unicode/utf8 69

62

quer ying packages 298

unsafe 354

POSIX file name 72

quot e ch arac ter, ' 56

package-le vel declarat ion 28

POSIX stand ard xi, 55, 72, 197

packages

precedence, operator 52, 63

race

building 293

precision

condit ion 21, 257, 258, 259

down loading 292

floating-p oint 56, 57, 63, 78

detec tor 271, 274

lo cat ing 291

numeric 55, 78

paradoxic al 267

quer ying 298

of con stants 78

randomize d test ing 307

palindrom e 191

predeclare d names, table of 28

range lo op 6, 9

palindrom e example 303, 305, 308

premature abstrac tion 216, 316, 317

range ov er channel 229

panic 64, 152, 253

premature opt imizat ion 324

range ov er map 94

panic built-in function 148, 149

Printf %% 10

range ov er str ing 69, 88

paradoxic al race 267

Printf verbs, table of 10

{{range}} temp late action 113

paral lel composition 224

Printf %b 10, 54, 75

raw str ing lit eral 66

paral lel, emb arrassingly 235

Printf %c 10, 56

re ach abi lit y 36

paral lelism 217

Printf %d 10, 55

re ad, stale 268

parameter

Printf %e 10, 57

re aders lock 266

... 91, 142, 143, 172

Printf %f 10, 57

re ad/w rit e mu tex 266, 267

func tion 119

Printf %g 10, 57

ReadWriteCloser interface 174

method receiver 156

Printf %[n] 56

ReadWriter interface 174

name 120

Printf %o 10, 55

real built-in function 61

passing 120

Printf %q 10, 56, 97

re ceive

unus ed 120

Printf %s 10

<-ch, channel 18, 225, 232

params example 348

Printf %*s 134

expression 225

params example package 348

Printf %T 10, 80, 83, 184, 331

non-blo cking channel 246

params.Unpack func tion 349

Printf %t 10, 83

ok value fro m ch annel 229

parent hes es 4, 6, 9, 52, 63, 119, 146,

Printf %#v 106, 207

re ceive-only channel typ e <-chan T

158, 285, 335, 345

Printf %v 10, 11

230

Parse example 152

Printf % x 71

re ceiver

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

INDEX

377

name, met hod 157

193

sexpr.Marshal func tion 340

ni l 159, 185

rune literal 56

sexpr.readList func tion 347

parameter, met hod 156

rune type 52, 67

sexpr.Unmarshal func tion 347

pointer 158, 167

rune slice to str ing conversion 71

SHA256 message digest 83

type, met hod 157

rune to str ing conversion 71

sha256 example 83

recover built-in function 152

runtime package 151

sh adowing declarat ion 46, 49, 206,

re cursion 121, 124, 247, 333, 339,

runt ime API 324

212

345, 359

runt ime sch edu ler 281

sh are d

re cursive

runtime.Stack func tion 151

lo ck 266

anonymou s func tion 137

var iables 257

data str ucture 101, 102, 107

satisfac tion, interface 171, 175

var iables, conc urrency wit h 257

type 48

Scal able Vec tor Graphics (SVG) 58

sh are d-memor y mu ltithre ading 217,

re defining new 35

sche duler, runtime 281

257

reference

Scheme programming langu age xiii

shif t op erator <<, lef t 54

call by 83

scop e

shif t op erator >>, rig ht 54

identity 87

de clarat ion 45, 137

short

type 9, 12, 93, 120

for 47

var iable decl arat ion 5, 7, 30, 31

reflect package 330

if-else 47

var iable decl arat ion scope 22, 48

reflec tion 329, 352, 359

label 46

var iable decl arat ion statement 7

reflect.StructTag type 348

lo cal variable 135

short-circuit

reflect.Type type 330

lo op var iable 141, 236

AND operator && 63

reflect.Type.Field method 348

pit fal l 140

evaluation 63

reflect.TypeOf func tion 330

short var iable decl arat ion 22, 48

OR operator || 63

reflect.Value type 331, 342

switch 47

sig nature, function 120

reflect.Value zero value 332

search algor it hm, bre adt h-firs t 139,

sig ned int eger 52, 54

reflect.Value.Addr method 342

239

s[i:j], slice operator 84, 86

reflect.Value.CanAddr method

search algor it hm, depth-firs t 136

s[i:j], subst ring operator 65, 86

342

SearchIssues func tion 111

simple statement 6, 22

reflect.Value.Interface method

select case 245

Sizeof table 354

ptg16091132

331, 342

select, default case in 246

sleep example 179

reflect.Value.Kind method 332

select, non-blo cking 246

slice 4

reflect.ValueOf func tion 331, 337

select st atement 244, 245

argument 86

reflect.Zero func tion 345

select{} st atement 245

capacity 88, 89

regexp package 149

sele ctive recov ery 152

capacity growt h di agram 90

regexp.Compile func tion 149

sele ctor, field 156

comp arison 87

regexp.MustCompile func tion 149

sele ctor, met hod 156

extending a 86

regu lar expression 66, 149, 305, 321

semaph ore , binary 262

ke y, map wit h 97

re lat ion, ‘‘happ ens before’’ 226, 257,

semaph ore , counting 241

literal 38, 86

261, 277

semicolon 3, 6

make 88, 322

re lat ions hip, ‘‘has a’’ 162

send ch<-, channel 18, 225, 232

ni l 87

re lat ions hip, ‘‘is a’’ 162, 175

send statement 225

of months diagram 84

remainder operator % 52, 166

send-only channel typ e chan<- T

op erator s[i:j] 84, 86

renaming imp ort 286

230

ro tat ion algor it hm 86

rendezvous 234

separate compi lat ion 284

te chniques, in-place 91

repl acement charac ter (, Unico de

sequence diag ram, thumbnail 238

type 84

70, 98

sequential consistenc y 268, 269

us ed as stack 123

repository, gopl.io xv i

serial confinement 262

zero lengt h 87

re quest

server example 19, 21

zero value 74, 87

HT TP GET 21, 127, 272, 348

server

SMTP client 312

HT TP POST 348

ch at 253

so cket

mu ltiplexer, HTTP 193

conc urrent clo ck 219

TCP 219

resu lt list, function 119

conc urrent ech o 222

UDP 219

resu lt, named 120, 126

set, map as 96, 202

Unix domain 219

return, bare 126

sett er met hod 169

sort algor it hm, top olog ical 136

return st atement 29, 120, 125

sexpr example 340

sort package 95, 186, 189

rev example 86

S-expression

sort.Float64s func tion 191

reverb example 223, 224

de code example 347

sort.Interface interface 186

right shif t op erator >> 54

de coding 344

sort.Ints func tion 191

Ru by programming langu age xv,

enco ding 338

sort.IntsAreSorted func tion 191

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

378

INDEX

sort.IntSlice type 191

strconv.ParseInt func tion 75

surface example 59, 203

sort.Reverse func tion 189

strconv.ParseUint func tion 75

surface figure, 3-D 58, 203

sort.Strings func tion 95, 137, 191

st ream, io.Discard 22

SVG 58

spinner example 218

String method 40, 166, 329

SWIG 361

Sprint func tion 330

st ring

Sw iss army knife 290

SQL API 211

conc atenation operator + 5, 65

switch, default case in 23

SQL inj e ction att ack 211

conv ersion 71

switch, initializat ion statement in

sqlQuote func tion 211, 212

immut abi lit y 65, 73

24

squares example 135

index operat ion 64

switch scop e 47

Sque ak, Ne wsque ak prog ramming

literal 65

switch st atement 23, 47

language xiii

literal, raw 66

switch st atement, tag less 24

st ack

range ov er 69, 88

switch st atement, typ e 210, 212,

al location 36

sh aring diagram 65

214, 329

fixe d-size 124

test 71

sw itch, context 280

growth 124, 280, 358

to byte slice conv ersion 40, 73

sync package 237, 263

implementation wit h slice 92, 215

to rune slice conv ersion 71, 88

sy nchro nou s ch annel 226

ov erflow 124

zero value 5, 7, 30

sync.Mutex type 263, 269

slice used as 123

comp arison 65

(*sync.Mutex).Lock method 21,

trace 149, 253

strings package 7, 71, 72, 289

146, 263

var iable 36

strings.Contains func tion 69

(*sync.Mutex).Unlock method 21,

var iable-size 124

strings.HasPrefix func tion 69

146, 263

st ale read 268

strings.HasSuffix func tion 69

sync.Once type 270

st and ard

strings.Index func tion 289

(*sync.Once).Do method 270

IEEE 754 56, 57

strings.Join func tion 7, 12

sync.RWMutex type 266, 270

POSIX xi, 55, 72, 197

strings.Map func tion 133

(*sync.RWMutex).RLock method

Unico de 2, 27, 52, 66, 67, 69, 97

strings.NewReader func tion 289

266

st atement

strings.NewReplacer func tion 289

(*sync.RWMutex).RUnlock method

--, decrement 5, 37

strings.Reader type 289

266

++, increment 5, 37, 94

strings.Replacer type 289

sync.WaitGroup type 237, 250, 274

ptg16091132

assig nment 5, 7, 36, 52, 94, 173

strings.Split func tion 12

(*sync.WaitGroup).Add method

break 24, 46

strings.ToLower func tion 72

238

continue 24, 46

strings.ToUpper func tion 72

(*sync.WaitGroup).Done method

coverage 318, 320

struct de clarat ion 99

238

defer 144, 150, 264

st ruc t

syscall package 196, 208

fallthrough 23, 212

comp arison 104

syscall.Errno type 196, 197

for 6

embedding 104, 161

system cal l API 196

go 18, 218

Employee 100

goto 24

empt y 102

table of

if-else 9, 22, 47

field 15, 99

binary operator s 52

label 46

field, anony mou s 104, 105, 106,

bit w ise operator s 53

labele d 46

162

comp arison operator s 53

return 29, 120, 125

field, emb edde d 161

es cap e sequences 66

select{} 245

field, exp ort of 101, 106, 109, 110,

ke yword s 27

select 244, 245

168

predeclare d names 28

send 225

field order 101, 355

Printf verbs 10

short var iable decl arat ion 7

field tag 109, 348

UTF-8 enco dings 67

simple 6, 22

hole 354

table, Sizeof 354

switch 23, 47

hole diag ram 355

table-dr iven testing 200, 306, 319

tagless switch 24

literal 15, 102, 106

tag, str uct field 109, 348

type switch 210, 212, 214, 329

literal, address of 103

tagless switch st atement 24

unreach able 120

pointer to 100, 103

tags, bui ld 296

storage example package 312, 313

type 15, 24, 99

TCP socket 219

St range lov e, Dr. 336

struct{} type 227, 241, 250

te chniques, in-place slice 91

st rateg ies, er ror-handling 128, 152,

st ruc t type, unnamed 163

tempconv example 39, 180, 289

310, 316

st ruc t zero value 102

tempconv example package 42

strconv package 22, 71, 75

su bst itutabi lit y 193

temp erature conversion example 29

strconv.Atoi func tion 22, 75

su bst ring operator s[i:j] 65, 86

tempflag example 181

strconv.FormatInt func tion 75

su btype poly mor phism 211

template API 115

strconv.Itoa func tion 75

sum example 142

temp late

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

INDEX

379

| in 113

token-b ased decoder API 213, 215,

method receiver 157

ac tion, {{range}} 113

347

mismatch 55

dot . in 113

token-b ased XML deco ding 213

name d 24, 39, 40, 105, 157

template.Funcs method 114

to ol

net.Conn 220

template.HTML type 116

cgo 361, 362

net.Listener 220

template.Must func tion 114

go 2, 42, 44, 290

numeric 51

template.New func tion 114

go doc 25

*os.File 11, 13, 172, 175, 185,

template.Parse method 114

godoc xv i, 25, 297, 326

336

Test func tion 302

gofmt 3, 4, 44, 286

os.FileInfo 247

test

goimports 3, 44, 286

os.LinkError 207

bl ack-b ox 310

golint 292

os.PathError 207

br ittle 317

topolog ical sor t algor it hm 136

re cursive 48

ch arac ter 71

topoSort example 136

reference 9, 12, 93, 120

coverage 318

trace example 146

reflect.StructTag 348

dep endency, cyc lic 314

trace, stack 149, 253

reflect.Type 330

echo 309

tree, binar y 102

reflect.Value 331, 342

fai lure message 306

treesort example 102

rune 52, 67

integrat ion 314

true boole an cons tant 63

slice 84

of word example 303

tr unc ation, floating-p oint 40, 55

sort.IntSlice 191

package, exter nal 285, 314

tuple assig nment 31, 37

strings.Reader 289

st ring 71

type de clarat ion 39

strings.Replacer 289

whit e-b ox 311

type ke yword 212

struct{} 227, 241, 250

assertion 306

type

st ruc t 15, 24, 99

_test.go file 285, 302, 303

abstrac t 24, 171

switch, case in 212

testing package 285, 302

ag gregate 81, 99

switch, default case in 212

test ing

ar ray 81

switch st atement 210, 212, 214,

a command 308

assertion 205, 211

329

randomize d 307

assertion, interface 208, 210

sync.Mutex 263, 269

table-dr iven 200, 306, 319

assertion, ok value fro m 206

sync.Once 270

ptg16091132

testing.B type 321

bool 63

sync.RWMutex 266, 270

testing.T type 302

bufio.Scanner 9

sync.WaitGroup 237, 250, 274

(*testing.T).Errorf method 200,

byte 52

syscall.Errno 196, 197

304, 306

bytes.Buffer 74, 169, 172, 185

template.HTML 116

(*testing.T).Fatal method 306

Celsius 39

testing.B 321

tests, writing effec tive 316, 317

chan 225

testing.T 302

text/scanner package 344

ch annel 18

time.Duration 76, 179

text/tabwriter package 188

<-chan T, receive-only channel

time.Time 114

text/template package 113, 300

230

uint 52

Thomps on, Ken xi, 67

chan<- T, send-only channel 230

uintptr 52, 354, 357

thre ad 218, 280

complex 61

underly ing 39

thre ad-lo cal storage 282

comp osite xv, 14, 81

unidirec tion al channel 230, 231

3-D sur face figure 58, 203

comp osition xv, 107, 162, 189

unnamed str uct 163

thumbnail example 236, 237, 238

concrete 24, 171, 211, 214

unsafe.Pointer 356

thumbnail example package 235

displaying met hods of a 351

url.URL 193

thumbnail sequence diag ram 238

empt y interface 176

types, untyped con stant 78

time package 18, 77, 183

error built-in 11, 128, 149, 196

time.After func tion 245

Fahrenheit 39

UDP socket 219

time.AfterFunc func tion 164

func tion 119, 120

uint type 52

time.Duration type 76, 179

http.HandlerFunc 194, 203

uintptr type 52, 354, 357

time.Minute cons tant 76

http.Request 21, 253

unar y op erator + 53

time.Now func tion 220

http.ResponseWriter 19, 22,

unar y op erator - 53

time.Parse func tion 220

191, 193

unbuffered channel 226

time.Second cons tant 164

int 52

undefine d behavior 260

time.Since func tion 114

interface{} 143, 176, 331

underly ing array 84, 88, 91, 187

time.Tick func tion 244, 246

interface 171, 174

underly ing typ e 39

time.Time type 114

interface dynamic 181

Unico de

time.Time.Format method 220

json.Decoder 111

co de point 67

title example 153

json.Encoder 111

es cap e 68, 107

title func tion 144, 145

map 9, 93

repl acement charac ter ( 70, 98

The Go Programming Language

© 2016 Alan A. A. Donovan & Brian W. Kernighan

revision 3b600c, date 29 Sep 2015

380

INDEX

st and ard 2, 27, 52, 66, 67, 69, 97

var iables, share d 257

unicode package 71

var iable-size stack 124

unicode.IsDigit func tion 71

var iadic function 142, 172

unicode.IsLetter func tion 71

ve ctor, bit 165

unicode.IsLower func tion 71

vendor ing 293

unicode.IsSpace func tion 93

visibi lit y 28, 29, 41, 168, 297

unicode.IsUpper func tion 71

visit func tion 122

unicode/utf8 package 69

unidirec tion al channel typ e 230, 231

wait example 130

union, discriminated 211, 213, 214

WaitForServer func tion 130

univers e block 46

walkDir func tion 247

Unix domain socket 219

we b

unmarsh aling JSON 110

craw ler 119

unnamed str uct typ e 163

craw ler, con cur rent 239

unnamed var iable 34, 88

framewor k 193

unreach able statement 120

while loop 6

unsafe package 354

whit e-b ox test 311

unsafe.AlignOf func tion 355

Wi l kes, Maur ice 301

unsafe.Offsetof func tion 355

Wirth, Nik laus xiii

unsafe.Pointer conv ersion 356

word example 303, 305, 308

unsafe.Pointer type 356

word example, test of 303

unsafe.Pointer zero value 356

worksp ace organizat ion 291

unsafe.Sizeof func tion 354

wr iter lock 266

unsig ned int eger 52, 54

wr iting effec tive tests 316, 317

untyped con stant typ es 78

unus ed parameter 120

xkcd JSON int erface 113

URL 123

XML deco ding 213

URL escape 111

XML (Extensible Marku p Language)

url.QueryEscape func tion 111

107

ptg16091132

url.URL type 193

(*xml.Decoder).Token method

urlvalues example 160

213

UTF-8 66, 67, 98

xmlselect example 215

UTF-8 enco dings, table of 67

utf8.DecodeRuneInString func tion

zero lengt h slice 87

69

zero value

utf8.RuneCountInString func tion

ar ray 82

69

boole an 30

utf8.UTFMax value 98

ch annel 225, 246

func tion 132

value

interface 182

addressable 32

map 95

call by 83, 120, 158

name d resu lt 120, 127

func tion 132

numb er 5, 30

interface 181

pointer 32

method 164

reflect.Value 332

utf8.UTFMax 98

slice 74, 87

var de clarat ion 5, 30

st ring 5, 7, 30

var iable

st ruc t 102

confinement 261

unsafe.Pointer 356

he ap 36

http.DefaultClient 253

io.Discard 18

io.EOF 132

lifet ime 35, 46, 135

lo cal 29, 141

os.Args 4

st ack 36

unnamed 34, 88

var iables, escaping 36

Join the

InformIt

AffiliAte teAm!

You love our titles and you love to

share them with your colleagues and friends...why

ptg16091132

not earn some $$ doing it!

If you have a website, blog, or even a Facebook

page, you can start earning money by putting

InformIT links on your page.

Whenever a visitor clicks on these links and makes

a purchase on informit.com, you earn commissions*

on all sales!

Every sale you bring to our site will earn you a

commission. All you have to do is post the links to

the titles you want, as many as you want, and we’ll

take care of the rest.

ApplY And get stArted!

It’s quick and easy to apply.

To learn more go to:

http://www.informit.com/affiliates/

*Valid for all books, eBooks and video sales at www.informit.com

Document Outline

Contents

Preface

The Origins of Go

The Go Project

Organization of the Book

Where to Find More Information

Acknowledgments

1. Tutorial

1.1. Hello, World

1.2. Command-Line Arguments

1.3. Finding Duplicate Lines

1.4. Animated GIFs

1.5. Fetching a URL

1.6. Fetching URLs Concurrently

1.7. A Web Server

1.8. Loose Ends

2. Program Structure

2.1. Names

2.2. Declarations

2.3. Variables

2.4. Assignments

2.5. Type Declarations

2.6. Packages and Files

2.7. Scope

3. Basic Data Types

3.1. Integers

3.2. Floating-Point Numbers

3.3. Complex Numbers

3.4. Booleans

3.5. Strings

3.6. Constants

4. Composite Types

4.1. Arrays

4.2. Slices

4.3. Maps

4.4. Structs

4.5. JSON

4.6. Text and HTML Templates

5. Functions

5.1. Function Declarations

5.2. Recursion

5.3. Multiple Return Values

5.4. Errors

5.5. Function Values

5.6. Anonymous Functions

5.7. Variadic Functions

5.8. Deferred Function Calls

5.9. Panic

5.10. Recover

6. Methods

6.1. Method Declarations

6.2. Methods with a Pointer Receiver

6.3. Composing Types by Struct Embedding

6.4. Method Values and Expressions

6.5. Example: Bit Vector Type

6.6. Encapsulation

7. Interfaces

7.1. Interfaces as Contracts

7.2. Interface Types

7.3. Interface Satisfaction

7.4. Parsing Flags with flag. Value

7.5. Interface Values

7.6. Sorting with sort. Interface

7.7. The http. Handler Interface

7.8. The error Interface

7.9. Example: Expression Evaluator

7.10. Type Assertions

7.11. Discriminating Errors with Type Assertions

7.12. Querying Behaviors with Interface Type Assertions

7.13. Type Switches

7.14. Example: Token-Based XML Decoding

7.15. A Few Words of Advice

8. Goroutines and Channels

8.1. Goroutines

8.2. Example: Concurrent Clock Server

8.3. Example: Concurrent Echo Server

8.4. Channels

8.5. Looping in Paralle

8.6. Example: Concurrent Web Crawler

8.7. Multiplexing with select

8.8. Example: Concurrent Directory Traversal

8 9 Cancellation

8.10. Example: Chat Server

9. Concurrency with Shared Variables

9.1. Race Conditions

9.2. Mutual Exclusion: sync. Mutex

9.3. Read/Write Mutexes: sync. RWMutex

9.4. Memory Synchronization

9.5. Lazy Initialization: sync.Once

9.6. The Race Detector

9.7. Example: Concurrent Non-Blocking Cache

9.8. Goroutinesand Threads

10. Packages and the Go Tool

10.1. Introduction

10.2. Import Paths

10.3. The Package Declaration

10.4. Import Declarations

10.5. Blank Imports

10.6. Packages and Naming

10.7. The Go Tool

11. Testing

11.1. The go test Tool

11.2. Test Functions

11.3. Coverage

11.4. Benchmark Functions

11.5. Profiling

11.6. Example Functions

12. Reflection

12.1. Why Reflection?

12.2. reflect. Type and reflect. Value

12.3. Display, a Recursive Value Printer

12.4. Example: Encoding S-Expressions

12.5. Setting Variables with reflect.Value

12.6. Example: Decoding S-Expressions

12.7. Accessing Struct Field Tags

12.8. Displaying the Methods of a Type

12.9. A Word of Caution

13. Low-Level Programming

13.1. unsafe. Sizeof, Alignof,and Offsetof

13.2. unsafe. Pointer

13.3. Example: Deep Equivalence

13.4. Calling C Code with cgo

13.5. Another Word of Caution

Index

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

T

U

V

W

X

Z